diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index c430287fa..60ad52ddb 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -82,6 +82,24 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - Data synchronization between IoTDB of 1. x series version and IoTDB of 2. x and above series versions is not supported. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. +- When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + + * The conversion rules for type mismatches are as follows: + + | Source Type | Target Type | Conversion Rule | + |---------------------|-------------|---------------------------------------------------------------------------------| + | Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | + | Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | + | BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | + | TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | + | TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | + | TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | + | TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | + | Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | + | DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | + | TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + + > **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. ## 2. Usage Instructions @@ -533,8 +551,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | | :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | | source | iotdb-source | String: iotdb-source | Yes | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | No | - | | mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | | mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | @@ -542,8 +560,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -554,39 +572,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:--------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | diff --git a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index c4dcd5b93..f62a412b3 100644 --- a/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -82,6 +82,26 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. - Object-type data export is supported since version V2.0.9.2. +- When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + + * The conversion rules for type mismatches are as follows: + + | Source Type | Target Type | Conversion Rule | + |---------------------|-------------|---------------------------------------------------------------------------------| + | Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | + | Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | + | BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | + | TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | + | TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | + | TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | + | TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | + | Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | + | DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | + | TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + + > **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. + + ## 2. Usage Instructions @@ -703,8 +723,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | | :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | | source | iotdb-source | String: iotdb-source | Yes | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data,schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data,schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | No | - | | mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | | mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | @@ -713,8 +733,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -725,57 +745,60 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| :------- | :------------ | -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | -| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes). | Boolean: true / false. True: can synchronize; False: cannot synchronize; | Optional | False | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes). | Boolean: true / false. True: can synchronize; False: cannot synchronize; | No | False | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-air-gap-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -| :--------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- |:---------------------------------------------| -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| air-gap.handshake-timeout-ms | The timeout duration for the handshake requests when the sender and receiver attempt to establish a connection for the first time, in milliseconds. | Integer | No | 5000 | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:--------------------------------------------| :----------------------------------------------------------- | :----------------------------------------------------------- | :------- |:---------------------------------------------| +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| air-gap.handshake-timeout-ms | The timeout duration for the handshake requests when the sender and receiver attempt to establish a connection for the first time, in milliseconds. | Integer | No | 5000 | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | -| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes).(Available since v2.0.5) | Boolean: true / false. True: can synchronize; False: cannot synchronize; | Optional | False | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:--------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes).(Available since v2.0.5) | Boolean: true / false. True: can synchronize; False: cannot synchronize; | No | False | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | @@ -784,11 +807,11 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | **Parameter** | **Description** | **value Range** | **Required** | **Default Value** | | ---------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |--------------| | sink | write-back-sink | String: write-back-sink | Yes | - | -| user/username | User used for write-back | String: username | Optional | root | -| password | Password used for write-back | String: password | Optional | root123 | -| user-id | User ID corresponding to the user | String | Optional | root | -| cli-hostname | CLI hostname corresponding to the user | String | Optional | root | -| use-event-user-name | Whether to use another user's username if the event contains one (generally not needed now because there is no external source) | Boolean: true / false | Optional | false | +| user/username | User used for write-back | String: username | No | root | +| password | Password used for write-back | String: password | No | root123 | +| user-id | User ID corresponding to the user | String | No | root | +| cli-hostname | CLI hostname corresponding to the user | String | No | root | +| use-event-user-name | Whether to use another user's username if the event contains one (generally not needed now because there is no external source) | Boolean: true / false | No | false | #### opc-ua-sink @@ -802,7 +825,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.opcua.enable-anonymous-access | Whether OPC UA allows anonymous access | Boolean | No | true | | sink.user | User for OPC UA, specified in the configuration | String | No | root | | sink.password | Password for OPC UA, specified in the configuration | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | Optional | "null" | +| sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | No | "null" | #### tsfile-local-sink diff --git a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md index cbaf5c2f6..5c641b2f3 100644 --- a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md @@ -105,6 +105,27 @@ By declaratively configuring the specific content of the three parts through SQL - Data under root.__system and root.__audit will not be synchronized. +3. Automatic Type Conversion for Pipe Sink + +When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + +The conversion rules for type mismatches are as follows: + +| Source Type | Target Type | Conversion Rule | +|---------------------|-------------|---------------------------------------------------------------------------------| +| Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | +| Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | +| BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | +| TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | +| TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | +| TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | +| TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | +| Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | +| DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | +| TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + +> **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. + ## 2. Usage Instructions Data synchronization tasks have three states: RUNNING, STOPPED, and DROPPED. The task state transitions are shown in the following diagram: @@ -556,20 +577,20 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| key | value | value range | required or not | default value | -| :------------------------------ |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| :------- | :------------- | -| source | iotdb-source | String: iotdb-source | Required | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | Optional | true | -| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | -| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true /false | Optional | true | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| Key | Value | Value range | Required | Default value | +|:-------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:---------|:---------------| +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | - | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | No | true | +| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | No | false | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | No | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true /false | No | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -581,39 +602,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-----------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-----------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | diff --git a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md index 62fb61d85..b3bf1debd 100644 --- a/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md @@ -107,6 +107,28 @@ By declaratively configuring the specific content of the three parts through SQL - Data under root.__system and root.__audit will not be synchronized. +3. Automatic Type Conversion for Pipe Sink + +When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + +The conversion rules for type mismatches are as follows: + +| Source Type | Target Type | Conversion Rule | +|---------------------|-------------|---------------------------------------------------------------------------------| +| Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | +| Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | +| BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | +| TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | +| TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | +| TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | +| TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | +| Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | +| DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | +| TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + +> **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. + + ## 2. Usage Instructions Data synchronization tasks have three states: RUNNING, STOPPED, and DROPPED. The task state transitions are shown in the following diagram: @@ -643,20 +665,20 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter（V1.3.3） -| key | value | value range | required or not | default value | -|:-------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:-----------------|:--------------| -| source | iotdb-source | String: iotdb-source | Required | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false |Optional | true | -| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | -| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8.2, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true / false | Optional | true | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| Key | Value | Value range | Required | Default value | +|:-------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:----------|:---------------| +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | - | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | No | true | +| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | No | false | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8.2, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | No | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true / false | No | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -667,53 +689,56 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- |:---------------------------------------------| -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- |:---------------------------------------------| +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-air-gap-sink -| key | value | value Range | required or not | Default Value | -| :--------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- |:---------------------------------------------| -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Required | - | -| node-urls | The URL of the data service port of any DataNode nodes on the target IoTDB | String. Example: ：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Required | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| air-gap.handshake-timeout-ms | The timeout duration of the handshake request when the sender and receiver first attempt to establish a connection, unit: ms | Integer | Optional | 5000 | +| Key | Value | Value Range | Required | Default Value | +|:----------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------|:---------------------------------------------| +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Required | - | +| node-urls | The URL of the data service port of any DataNode nodes on the target IoTDB | String. Example: ：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Required | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| air-gap.handshake-timeout-ms | The timeout duration of the handshake request when the sender and receiver first attempt to establish a connection, unit: ms | Integer | No | 5000 | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------|:---------------------------------------------| -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------|:---------------------------------------------| +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index c430287fa..60ad52ddb 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -82,6 +82,24 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - Data synchronization between IoTDB of 1. x series version and IoTDB of 2. x and above series versions is not supported. - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. +- When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + + * The conversion rules for type mismatches are as follows: + + | Source Type | Target Type | Conversion Rule | + |---------------------|-------------|---------------------------------------------------------------------------------| + | Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | + | Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | + | BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | + | TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | + | TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | + | TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | + | TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | + | Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | + | DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | + | TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + + > **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. ## 2. Usage Instructions @@ -533,8 +551,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | | :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | | source | iotdb-source | String: iotdb-source | Yes | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | No | - | | mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | | mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | @@ -542,8 +560,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | table-name | When the user connects with `sql_dialect` set to `table`, this parameter can be specified. Determines the scope of data capture, affecting the `data` in `inclusion`. Specifies the table name to filter. It can be a specific table name or a Java-style regular expression to match multiple tables. By default, all tables are matched. | String: Data table name or data table regular expression pattern string, which can be uncreated or non - existent tables. | No | ".*" | | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -554,39 +572,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:--------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | diff --git a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index c4dcd5b93..f62a412b3 100644 --- a/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -82,6 +82,26 @@ By declaratively configuring these three parts in an SQL statement, flexible dat - When performing data synchronization tasks, avoid executing any deletion operations to prevent inconsistencies between the two ends. - The `pipe` and `pipe plugins` for tree modes and table modes are designed to be isolated from each other. Before creating a `pipe`, it is recommended to first use the `show` command to query the built-in plugins available under the current `-sql_dialect` parameter configuration to ensure syntax compatibility and functional support. - Object-type data export is supported since version V2.0.9.2. +- When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + + * The conversion rules for type mismatches are as follows: + + | Source Type | Target Type | Conversion Rule | + |---------------------|-------------|---------------------------------------------------------------------------------| + | Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | + | Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | + | BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | + | TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | + | TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | + | TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | + | TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | + | Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | + | DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | + | TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + + > **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. + + ## 2. Usage Instructions @@ -703,8 +723,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | **Parameter** | **Description** | **Value Range** | **Required** | **Default Value** | | :----------------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| :----------- | :---------------------------------------------------------- | | source | iotdb-source | String: iotdb-source | Yes | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data,schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | Optional | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data,schema and auth | String:all, data(insert,delete), schema(database,table,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,table,ttl), auth | No | - | | mode.streaming | This parameter specifies the source of time-series data capture. It applies to scenarios where `mode.streaming` is set to `false`, determining the capture source for `data.insert` in `inclusion`. Two capture strategies are available: - **true**: Dynamically selects the capture type. The system adapts to downstream processing speed, choosing between capturing each write request or only capturing TsFile file sealing requests. When downstream processing is fast, write requests are prioritized to reduce latency; when processing is slow, only file sealing requests are captured to prevent processing backlogs. This mode suits most scenarios, optimizing the balance between processing latency and throughput. - **false**: Uses a fixed batch capture approach, capturing only TsFile file sealing requests. This mode is suitable for resource-constrained applications, reducing system load. **Note**: Snapshot data captured when the pipe starts will only be provided for downstream processing as files. | Boolean: true / false | No | true | | mode.strict | Determines whether to strictly filter data when using the `time`, `path`, `database-name`, or `table-name` parameters: - **true**: Strict filtering. The system will strictly filter captured data according to the specified conditions, ensuring that only matching data is selected. - **false**: Non-strict filtering. Some extra data may be included during the selection process to optimize performance and reduce CPU and I/O consumption. | Boolean: true / false | No | true | | mode.snapshot | This parameter determines the data capture mode, affecting the `data` in `inclusion`. Two modes are available: - **true**: Static data capture. A one-time data snapshot is taken when the pipe starts. Once the snapshot data is fully consumed, the pipe automatically terminates (executing `DROP PIPE` SQL automatically). - **false**: Dynamic data capture. In addition to capturing snapshot data when the pipe starts, it continuously captures subsequent data changes. The pipe remains active to process the dynamic data stream. | Boolean: true / false | No | false | @@ -713,8 +733,8 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | start-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **greater than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | end-time | Determines the scope of data capture, affecting the `data` in `inclusion`. Data with an event time **less than or equal to** this parameter will be selected for stream processing in the pipe. | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp)orString: ISO format timestamp supported by IoTDB | No | Long: [Long.MIN_VALUE, Long.MAX_VALUE](Unix bare timestamp) | | mode.double-living | Whether to enable full dual-active mode. When enabled, the system will ignore the `-sql_dialect` connection method to capture all tree-table model data and not forward data synced from another pipe (to avoid circular synchronization). | Boolean: true / false | No | false | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -725,57 +745,60 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| :------- | :------------ | -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | -| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes). | Boolean: true / false. True: can synchronize; False: cannot synchronize; | Optional | False | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes). | Boolean: true / false. True: can synchronize; False: cannot synchronize; | No | False | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-air-gap-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -| :--------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- |:---------------------------------------------| -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| air-gap.handshake-timeout-ms | The timeout duration for the handshake requests when the sender and receiver attempt to establish a connection for the first time, in milliseconds. | Integer | No | 5000 | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:--------------------------------------------| :----------------------------------------------------------- | :----------------------------------------------------------- | :------- |:---------------------------------------------| +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| air-gap.handshake-timeout-ms | The timeout duration for the handshake requests when the sender and receiver attempt to establish a connection for the first time, in milliseconds. | Integer | No | 5000 | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | -| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes).(Available since v2.0.5) | Boolean: true / false. True: can synchronize; False: cannot synchronize; | Optional | False | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:--------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/usename | Usename for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| mark-as-general-write-request | This parameter controls whether data forwarded by external pipes can be synchronized between dual-active pipes (configured on the sender side of dual-active external pipes).(Available since v2.0.5) | Boolean: true / false. True: can synchronize; False: cannot synchronize; | No | False | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | @@ -784,11 +807,11 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | **Parameter** | **Description** | **value Range** | **Required** | **Default Value** | | ---------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |--------------| | sink | write-back-sink | String: write-back-sink | Yes | - | -| user/username | User used for write-back | String: username | Optional | root | -| password | Password used for write-back | String: password | Optional | root123 | -| user-id | User ID corresponding to the user | String | Optional | root | -| cli-hostname | CLI hostname corresponding to the user | String | Optional | root | -| use-event-user-name | Whether to use another user's username if the event contains one (generally not needed now because there is no external source) | Boolean: true / false | Optional | false | +| user/username | User used for write-back | String: username | No | root | +| password | Password used for write-back | String: password | No | root123 | +| user-id | User ID corresponding to the user | String | No | root | +| cli-hostname | CLI hostname corresponding to the user | String | No | root | +| use-event-user-name | Whether to use another user's username if the event contains one (generally not needed now because there is no external source) | Boolean: true / false | No | false | #### opc-ua-sink @@ -802,7 +825,7 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 | sink.opcua.enable-anonymous-access | Whether OPC UA allows anonymous access | Boolean | No | true | | sink.user | User for OPC UA, specified in the configuration | String | No | root | | sink.password | Password for OPC UA, specified in the configuration | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | Optional | "null" | +| sink.opcua.placeholder | A placeholder string used to substitute for null mapping paths when the value of the ID column is null | String | No | "null" | #### tsfile-local-sink diff --git a/src/UserGuide/latest/User-Manual/Data-Sync_apache.md b/src/UserGuide/latest/User-Manual/Data-Sync_apache.md index cbaf5c2f6..5c641b2f3 100644 --- a/src/UserGuide/latest/User-Manual/Data-Sync_apache.md +++ b/src/UserGuide/latest/User-Manual/Data-Sync_apache.md @@ -105,6 +105,27 @@ By declaratively configuring the specific content of the three parts through SQL - Data under root.__system and root.__audit will not be synchronized. +3. Automatic Type Conversion for Pipe Sink + +When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + +The conversion rules for type mismatches are as follows: + +| Source Type | Target Type | Conversion Rule | +|---------------------|-------------|---------------------------------------------------------------------------------| +| Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | +| Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | +| BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | +| TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | +| TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | +| TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | +| TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | +| Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | +| DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | +| TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + +> **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. + ## 2. Usage Instructions Data synchronization tasks have three states: RUNNING, STOPPED, and DROPPED. The task state transitions are shown in the following diagram: @@ -556,20 +577,20 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter -| key | value | value range | required or not | default value | -| :------------------------------ |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| :------- | :------------- | -| source | iotdb-source | String: iotdb-source | Required | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | Optional | true | -| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | -| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true /false | Optional | true | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| Key | Value | Value range | Required | Default value | +|:-------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:---------|:---------------| +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | - | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | No | true | +| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | No | false | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | No | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true /false | No | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -581,39 +602,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-----------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- | :------------ | +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | root | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-----------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------| :------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | root | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | diff --git a/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md b/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md index 62fb61d85..b3bf1debd 100644 --- a/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md +++ b/src/UserGuide/latest/User-Manual/Data-Sync_timecho.md @@ -107,6 +107,28 @@ By declaratively configuring the specific content of the three parts through SQL - Data under root.__system and root.__audit will not be synchronized. +3. Automatic Type Conversion for Pipe Sink + +When Pipe fails to write data to the sink due to field type mismatches, IoTDB automatically converts the data to the field types defined in the existing sink schema and retries the write operation to improve synchronization success rate. This feature is controlled by the parameter `sink.exception.data.convert-on-type-mismatch`. Refer to the subsequent sink parameter table for detailed parameter descriptions. + +The conversion rules for type mismatches are as follows: + +| Source Type | Target Type | Conversion Rule | +|---------------------|-------------|---------------------------------------------------------------------------------| +| Numeric Type | Numeric Type| Convert to the target numeric type. Truncation, precision loss or overflow may occur. | +| Numeric Type | BOOLEAN | `0` is converted to `false`; non-zero values are converted to `true`. | +| BOOLEAN | Numeric Type| `true` is converted to `1`; `false` is converted to `0`. | +| TEXT, STRING, BLOB | BOOLEAN | Parse the string into a BOOLEAN value. | +| TEXT, STRING, BLOB | Numeric Type| Parse the string into the target numeric type. If parsing fails, write the default value `0`, `0L` or `0.0`. | +| TEXT, STRING, BLOB | TIMESTAMP | Parse the string into a TIMESTAMP value. If parsing fails, write the default value `0L`. | +| TEXT, STRING, BLOB | DATE | Parse the string into a DATE value. If parsing fails, write the default date `1970-01-01`. | +| Invalid Numeric Value | DATE | If conversion to a valid DATE fails, write the default date `1970-01-01`. | +| DATE | TIMESTAMP | Convert to the timestamp of 00:00 (UTC) on the same day. | +| TIMESTAMP | DATE | Convert to the corresponding date in UTC. | + +> **Note**: Automatic conversion is performed based on the existing sink schema and will **not** modify the sink schema. This feature prioritizes continuous data synchronization, which may result in precision loss or writing of default values. + + ## 2. Usage Instructions Data synchronization tasks have three states: RUNNING, STOPPED, and DROPPED. The task state transitions are shown in the following diagram: @@ -643,20 +665,20 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 ### 5.1 source parameter（V1.3.3） -| key | value | value range | required or not | default value | -|:-------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:-----------------|:--------------| -| source | iotdb-source | String: iotdb-source | Required | - | -| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | data.insert | -| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | Optional | - | -| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false |Optional | true | -| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | Optional | true | -| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | Optional | false | -| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8.2, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | Optional | root.** | -| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MIN_VALUE | -| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | Optional | Long.MAX_VALUE | -| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true / false | Optional | true | -| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | Optional | false | -| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | Optional | no-privileges | +| Key | Value | Value range | Required | Default value | +|:-------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------|:----------|:---------------| +| source | iotdb-source | String: iotdb-source | Yes | - | +| inclusion | Used to specify the range of data to be synchronized in the data synchronization task, including data, schema, and auth | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | data.insert | +| inclusion.exclusion | Used to exclude specific operations from the range specified by inclusion, reducing the amount of data synchronized | String:all, data(insert,delete), schema(database,timeseries,ttl), auth | No | - | +| mode.streaming | Specifies the capture source for time-series data writes. Applicable when mode.streamingis false, determining the source for capturing data.insertspecified in inclusion. Offers two strategies:- true: ​​Dynamic capture selection.​​ The system adaptively chooses between capturing individual write requests or only TsFile sealing requests based on downstream processing speed. Prioritizes capturing write requests for lower latency when processing is fast; captures only file sealing requests to avoid backlog when slow. Suitable for most scenarios, balancing latency and throughput optimally.- false: ​​Fixed batch capture.​​ Captures only TsFile sealing requests. Suitable for resource-constrained scenarios to reduce system load. Note: The snapshot data captured upon pipe startup is only provided to downstream processing in file format. | Boolean: true / false | No | true | +| mode.strict | Determines the strictness when filtering data using time/ path/ database-name/ table-nameparameters:- true: ​​Strict filtering.​​ The system strictly filters captured data according to the given conditions, ensuring only matching data is selected.- false: ​​Non-strict filtering.​​ The system may include some extra data during filtering. Suitable for performance-sensitive scenarios to reduce CPU and I/O consumption. | Boolean: true / false | No | true | +| mode.snapshot | Determines the capture mode for time-series data, affecting the dataspecified in inclusion. Offers two modes:- true: ​​Static data capture.​​ Upon pipe startup, a one-time data snapshot is captured. ​​The pipe will automatically terminate (DROP PIPE SQL is executed automatically) after the snapshot data is fully consumed.​​- false: ​​Dynamic data capture.​​ In addition to capturing a snapshot upon startup, the pipe continuously captures subsequent data changes. The pipe runs continuously to handle the dynamic data stream. | Boolean: true / false | No | false | +| path | Can be specified when the user connects with sql_dialectset to tree. For upgraded user pipes, the default sql_dialectis tree. This parameter determines the capture scope for time-series data, affecting the dataspecified in inclusion, as well as some sequence-related metadata. Data is selected into the streaming pipe if its tree model path matches the specified path.
Starting from version V2.0.8.2, this parameter supports specifying multiple exact paths in a single pipe, e.g., `'path'='root.test.d0.s1,root.test.d0.s2,root.test.d0.s3'`. | String: IoTDB-standard tree path pattern, wildcards allowed | No | root.** | +| start-time | The start event time for synchronizing all data, including start-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MIN_VALUE | +| end-time | The end event time for synchronizing all data, including end-time | Long: [Long.MIN_VALUE, Long.MAX_VALUE] | No | Long.MAX_VALUE | +| forwarding-pipe-requests | Whether to forward data written by other Pipes (usually data synchronization) | Boolean: true / false | No | true | +| mods | Same as mods.enable, whether to send the MODS file for TSFile. | Boolean: true / false | No | false | +| skipIf | Which errors can be skipped? Currently only the insufficient privileges error. | String:no-privileges | No | no-privileges | > 💎 **Note:** The difference between the values of true and false for the data extraction mode `mode.streaming` > @@ -667,53 +689,56 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- |:---------------------------------------------| -| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| :----------------------------------------------------------- | :------- |:---------------------------------------------| +| sink | iotdb-thrift-sink or iotdb-thrift-async-sink | String: iotdb-thrift-sink or iotdb-thrift-async-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | The maximum number of bytes allowed to be transmitted per second. The compressed bytes (such as after compression) are calculated. If it is less than 0, there is no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-air-gap-sink -| key | value | value Range | required or not | Default Value | -| :--------------------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | :------- |:---------------------------------------------| -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Required | - | -| node-urls | The URL of the data service port of any DataNode nodes on the target IoTDB | String. Example: ：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Required | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| air-gap.handshake-timeout-ms | The timeout duration of the handshake request when the sender and receiver first attempt to establish a connection, unit: ms | Integer | Optional | 5000 | +| Key | Value | Value Range | Required | Default Value | +|:----------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------|:---------------------------------------------| +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | Required | - | +| node-urls | The URL of the data service port of any DataNode nodes on the target IoTDB | String. Example: ：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Required | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| air-gap.handshake-timeout-ms | The timeout duration of the handshake request when the sender and receiver first attempt to establish a connection, unit: ms | Integer | No | 5000 | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | #### iotdb-thrift-ssl-sink -| **Parameter** | **Description** | Value Range | Required | Default Value | -|:----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------|:---------------------------------------------| -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | -| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | -| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | -| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | -| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | -| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | -| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | -| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | -| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | -| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | -| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | -| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | -| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | -| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | -| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| **Parameter** | **Description** | Value Range | Required | Default Value | +|:-------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------|:---------------------------------------------| +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | Yes | - | +| node-urls | URLs of the DataNode service ports on the target IoTDB. (please note that the synchronization task does not support forwarding to its own service). | String. Example：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | Yes | - | +| user/username | Username for connecting to the target IoTDB. Must have appropriate permissions. | String | No | root | +| password | Password for the username. | String | No | TimechoDB@2021 (Before V2.0.6.x it is root) | +| batch.enable | Enables batch mode for log transmission to improve throughput and reduce IOPS. | Boolean: true, false | No | true | +| batch.max-delay-seconds | Maximum delay (in seconds) for batch transmission. | Integer | No | 1 | +| batch.max-delay-ms | Maximum delay (in ms) for batch transmission. (Available since v2.0.5) | Integer | No | 1 | +| batch.size-bytes | Maximum batch size (in bytes) for batch transmission. | Long | No | 16*1024*1024 | +| compressor | The selected RPC compression algorithm. Multiple algorithms can be configured and will be adopted in sequence for each request. | String: snappy / gzip / lz4 / zstd / lzma2 | No | "" | +| compressor.zstd.level | When the selected RPC compression algorithm is zstd, this parameter can be used to additionally configure the compression level of the zstd algorithm. | Int: [-131072, 22] | No | 3 | +| rate-limit-bytes-per-second | Maximum bytes allowed per second for transmission (calculated after compression). Set to a value less than 0 for no limit. | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | No | -1 | +| load-tsfile-strategy | When synchronizing file data, ​​whether the receiver waits for the local load tsfile operation to complete before responding to the sender​​:
​​sync​​: Wait for the local load tsfile operation to complete before returning the response.
​​async​​: Do not wait for the local load tsfile operation to complete; return the response immediately. | String: sync / async | No | sync | +| ssl.trust-store-path | Path to the trust store certificate for SSL connection. | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | Yes | - | +| ssl.trust-store-pwd | Password for the trust store certificate. | Integer | Yes | - | +| format | The payload formats for data transmission include the following options:
- hybrid: The format depends on what is passed from the processor (either tsfile or tablet), and the sink performs no conversion.
- tsfile: Data is forcibly converted to tsfile format before transmission. This is suitable for scenarios like data file backup.
- tablet: Data is forcibly converted to tsfile format before transmission. This is useful for data synchronization when the sender and receiver have incompatible data types (to minimize errors). | String: hybrid / tsfile / tablet | No | hybrid | +| exception.data.convert-on-type-mismatch | Whether to enable automatic conversion when data types mismatch on the sink side | Boolean: true / false | No | true | diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md index 6a098bfcb..7e1f4dd61 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md @@ -81,7 +81,26 @@ - 不支持 1.x 系列版本 IoTDB 与 2.x 以及以上系列版本的 IoTDB 之间进行数据同步。 - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 - +- 当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + + * 类型不匹配时的转换规则如下： + + | 源类型 | 目标类型 | 转换规则 | + | -------------------- | ----------- | -------------------------------------------------------------------------------- | + | 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | + | 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | + | BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | + | TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | + | TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | + | TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | + | TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | + | 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | + | DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | + | TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + + > 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 + + ## 2. 使用说明 数据同步任务有三种状态：RUNNING、STOPPED 和 DROPPED。任务状态转换如下图所示： @@ -544,39 +563,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms） (V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms） (V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | diff --git a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md index b72f10e10..e05d9c999 100644 --- a/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/Master/Table/User-Manual/Data-Sync_timecho.md @@ -82,6 +82,24 @@ - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 - 自 V2.0.9.2 版本起支持 Object 类型数据导出。 +- 当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + + * 类型不匹配时的转换规则如下： + + | 源类型 | 目标类型 | 转换规则 | + | -------------------- | ----------- | -------------------------------------------------------------------------------- | + | 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | + | 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | + | BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | + | TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | + | TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | + | TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | + | TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | + | 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | + | DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | + | TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + + > 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 ## 2. 使用说明 @@ -720,59 +738,62 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/usename | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | -| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端）(V2.0.5及以后版本支持） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/usename | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端）(V2.0.5及以后版本支持） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-air-gap-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- | -------- | -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- | -------- | +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | -| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### write-back-sink diff --git a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md index 42622153d..0f81dd0a2 100644 --- a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md @@ -106,6 +106,28 @@ - 对于 root.__system, root.__audit 均不会同步 +3. Pipe 接收端类型自动转换 + +当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + +类型不匹配时的转换规则如下： + +| 源类型 | 目标类型 | 转换规则 | +| -------------------- | ----------- | -------------------------------------------------------------------------------- | +| 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | +| 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | +| BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | +| TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | +| TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | +| TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | +| TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | +| 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | +| DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | +| TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + +> 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 + + ## 2. 使用说明 数据同步任务有三种状态：RUNNING、STOPPED 和 DROPPED。任务状态转换如下图所示： @@ -578,39 +600,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | diff --git a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md index 893014399..c4cfe8fa7 100644 --- a/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/Master/Tree/User-Manual/Data-Sync_timecho.md @@ -108,6 +108,29 @@ - 对于 root.__system, root.__audit 均不会同步 +3. Pipe 接收端类型自动转换 + +当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + +类型不匹配时的转换规则如下： + +| 源类型 | 目标类型 | 转换规则 | +| -------------------- | ----------- | -------------------------------------------------------------------------------- | +| 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | +| 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | +| BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | +| TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | +| TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | +| TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | +| TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | +| 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | +| DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | +| TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + +> 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 + + + ## 2. 使用说明 数据同步任务有三种状态：RUNNING、STOPPED 和 DROPPED。任务状态转换如下图所示： @@ -665,53 +688,56 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| key | value | value 取值范围 | 是否必填 | 默认取值 | -|-----------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |----------------------------------| -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| key | value | value 取值范围 | 是否必填 | 默认取值 | +|-------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |----------------------------------| +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-air-gap-sink -| key | value | value 取值范围 | 是否必填 | 默认取值 | -| ---------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| key | value | value 取值范围 | 是否必填 | 默认取值 | +|-------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| key | value | value 取值范围 | 是否必填 | 默认取值 | -| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| key | value | value 取值范围 | 是否必填 | 默认取值 | +|--------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md index 6a098bfcb..7e1f4dd61 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_apache.md @@ -81,7 +81,26 @@ - 不支持 1.x 系列版本 IoTDB 与 2.x 以及以上系列版本的 IoTDB 之间进行数据同步。 - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 - +- 当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + + * 类型不匹配时的转换规则如下： + + | 源类型 | 目标类型 | 转换规则 | + | -------------------- | ----------- | -------------------------------------------------------------------------------- | + | 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | + | 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | + | BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | + | TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | + | TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | + | TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | + | TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | + | 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | + | DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | + | TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + + > 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 + + ## 2. 使用说明 数据同步任务有三种状态：RUNNING、STOPPED 和 DROPPED。任务状态转换如下图所示： @@ -544,39 +563,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms） (V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms） (V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | diff --git a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md index b72f10e10..e05d9c999 100644 --- a/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/latest-Table/User-Manual/Data-Sync_timecho.md @@ -82,6 +82,24 @@ - 在进行数据同步任务时，请避免执行任何删除操作，防止两端状态不一致。 - 树模型与表模型的`pipe`及`pipe plugins`在设计上相互隔离，建议在创建`pipe`前先通过`show`命令查询当前`-sql_dialect`参数配置下可用的内置插件，以确保语法兼容性和功能支持。 - 自 V2.0.9.2 版本起支持 Object 类型数据导出。 +- 当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + + * 类型不匹配时的转换规则如下： + + | 源类型 | 目标类型 | 转换规则 | + | -------------------- | ----------- | -------------------------------------------------------------------------------- | + | 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | + | 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | + | BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | + | TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | + | TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | + | TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | + | TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | + | 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | + | DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | + | TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + + > 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 ## 2. 使用说明 @@ -720,59 +738,62 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/usename | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | -| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端）(V2.0.5及以后版本支持） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/usename | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端）(V2.0.5及以后版本支持） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-air-gap-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- | -------- | -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- | -------- | +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | -| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|--------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021,V2.0.6.x之前为 root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| mark-as-general-write-request | 该参数可控制双活 pipe 之间能否同步外部 pipe 转发的数据（配置到双活外部 pipe 的发送端） | Boolean: true / false。True：能同步；False：不能同步； | 选填 | False | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### write-back-sink diff --git a/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md b/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md index 42622153d..0f81dd0a2 100644 --- a/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md +++ b/src/zh/UserGuide/latest/User-Manual/Data-Sync_apache.md @@ -106,6 +106,28 @@ - 对于 root.__system, root.__audit 均不会同步 +3. Pipe 接收端类型自动转换 + +当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + +类型不匹配时的转换规则如下： + +| 源类型 | 目标类型 | 转换规则 | +| -------------------- | ----------- | -------------------------------------------------------------------------------- | +| 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | +| 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | +| BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | +| TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | +| TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | +| TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | +| TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | +| 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | +| DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | +| TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + +> 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 + + ## 2. 使用说明 数据同步任务有三种状态：RUNNING、STOPPED 和 DROPPED。任务状态转换如下图所示： @@ -578,39 +600,41 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|----------|--------------| +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | -|-----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| **参数** | **描述** | **value 取值范围** | **是否必填** | **默认取值** | +|------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| -------- | ------------ | +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | diff --git a/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md b/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md index 893014399..c4cfe8fa7 100644 --- a/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md +++ b/src/zh/UserGuide/latest/User-Manual/Data-Sync_timecho.md @@ -108,6 +108,29 @@ - 对于 root.__system, root.__audit 均不会同步 +3. Pipe 接收端类型自动转换 + +当 Pipe 向接收端写入数据因字段类型不匹配而失败时，IoTDB 可按照目标端已有 schema 的字段类型对数据进行转换，并重试写入，以提高同步成功率。该能力由 `sink.exception.data.convert-on-type-mismatch` 控制，参数说明见后文 sink 参数表。 + +类型不匹配时的转换规则如下： + +| 源类型 | 目标类型 | 转换规则 | +| -------------------- | ----------- | -------------------------------------------------------------------------------- | +| 数值类型 | 数值类型 | 按目标数值类型进行转换，可能发生截断、精度损失或溢出。 | +| 数值类型 | BOOLEAN | `0`转换为`false`，非`0`转换为`true`。 | +| BOOLEAN | 数值类型 | `true`转换为`1`，`false`转换为`0`。 | +| TEXT、STRING、BLOB | BOOLEAN | 按字符串解析为 BOOLEAN。 | +| TEXT、STRING、BLOB | 数值类型 | 按字符串解析为目标数值类型；解析失败时写入默认值`0`、`0L`或`0.0`。 | +| TEXT、STRING、BLOB | TIMESTAMP | 按字符串解析为 TIMESTAMP；解析失败时写入默认值`0L`。 | +| TEXT、STRING、BLOB | DATE | 按字符串解析为 DATE；解析失败时写入默认日期`1970-01-01`。 | +| 非法数值 | DATE | 若无法转换为合法 DATE，则写入默认日期`1970-01-01`。 | +| DATE | TIMESTAMP | 按 UTC 转换为当天零点对应的时间戳。 | +| TIMESTAMP | DATE | 按 UTC 转换为对应日期。 | + +> 注意：自动转换基于目标端已有 schema 执行，不会自动修改目标端 schema；该能力优先保证同步继续进行，可能导致精度损失或默认值写入。 + + + ## 2. 使用说明 数据同步任务有三种状态：RUNNING、STOPPED 和 DROPPED。任务状态转换如下图所示： @@ -665,53 +688,56 @@ pipe_all_sinks_rate_limit_bytes_per_second=-1 #### iotdb-thrift-sink -| key | value | value 取值范围 | 是否必填 | 默认取值 | -|-----------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |----------------------------------| -| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| key | value | value 取值范围 | 是否必填 | 默认取值 | +|-------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |----------------------------------| +| sink | iotdb-thrift-sink 或 iotdb-thrift-async-sink | String: iotdb-thrift-sink 或 iotdb-thrift-async-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-air-gap-sink -| key | value | value 取值范围 | 是否必填 | 默认取值 | -| ---------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| -| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| key | value | value 取值范围 | 是否必填 | 默认取值 | +|-------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| +| sink | iotdb-air-gap-sink | String: iotdb-air-gap-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| air-gap.handshake-timeout-ms | 发送端与接收端在首次尝试建立连接时握手请求的超时时长，单位：毫秒 | Integer | 选填 | 5000 | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true | #### iotdb-thrift-ssl-sink -| key | value | value 取值范围 | 是否必填 | 默认取值 | -| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| -| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | -| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | -| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | -| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | -| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | -| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | -| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | -| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | -| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | -| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | -| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | -| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | -| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | -| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | -| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| key | value | value 取值范围 | 是否必填 | 默认取值 | +|--------------------------------------------| ------------------------------------------------------------ | ------------------------------------------------------------ | -------- |-----------------------------------| +| sink | iotdb-thrift-ssl-sink | String: iotdb-thrift-ssl-sink | 必填 | - | +| node-urls | 目标端 IoTDB 任意多个 DataNode 节点的数据服务端口的 url（请注意同步任务不支持向自身服务进行转发） | String. 例：'127.0.0.1：6667，127.0.0.1：6668，127.0.0.1：6669'， '127.0.0.1：6667' | 必填 | - | +| user/username | 连接接收端使用的用户名，同步要求该用户具备相应的操作权限 | String | 选填 | root | +| password | 连接接收端使用的用户名对应的密码，同步要求该用户具备相应的操作权限 | String | 选填 | TimechoDB@2021, V2.0.6.x 之前为root | +| batch.enable | 是否开启日志攒批发送模式，用于提高传输吞吐，降低 IOPS | Boolean: true, false | 选填 | true | +| batch.max-delay-seconds | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：s） | Integer | 选填 | 1 | +| batch.max-delay-ms | 在开启日志攒批发送模式时生效，表示一批数据在发送前的最长等待时间（单位：ms）(V2.0.5及以后版本支持） | Integer | 选填 | 1 | +| batch.size-bytes | 在开启日志攒批发送模式时生效，表示一批数据最大的攒批大小（单位：byte） | Long | 选填 | 16*1024*1024 | +| compressor | 所选取的 rpc 压缩算法，可配置多个，对每个请求顺序采用 | String: snappy / gzip / lz4 / zstd / lzma2 | 选填 | "" | +| compressor.zstd.level | 所选取的 rpc 压缩算法为 zstd 时，可使用该参数额外配置 zstd 算法的压缩等级 | Int: [-131072, 22] | 选填 | 3 | +| rate-limit-bytes-per-second | 每秒最大允许传输的 byte 数，计算压缩后的 byte（如压缩），若小于 0 则不限制 | Double: [Double.MIN_VALUE, Double.MAX_VALUE] | 选填 | -1 | +| load-tsfile-strategy | 文件同步数据时，接收端请求返回发送端前，是否等待接收端本地的 load tsfile 执行结果返回。
sync：等待本地的 load tsfile 执行结果返回；
async：不等待本地的 load tsfile 执行结果返回。 | String: sync / async | 选填 | sync | +| ssl.trust-store-path | 连接目标端 DataNode 所需的 trust store 证书路径 | String.Example: '127.0.0.1:6667,127.0.0.1:6668,127.0.0.1:6669', '127.0.0.1:6667' | 必填 | - | +| ssl.trust-store-pwd | 连接目标端 DataNode 所需的 trust store 证书密码 | Integer | 必填 | - | +| format | 数据传输的payload格式， 可选项包括：
- hybrid： 取决于 processor 传递过来的格式（tsfile或tablet），sink不做任何转换。
- tsfile：强制转换成tsfile发送，可用于数据文件备份等场景。
- tablet：强制转换成tsfile发送，可用于发送端/接收端数据类型不完全兼容时的数据同步（以减少报错）。 | String: hybrid / tsfile / tablet | 选填 | hybrid | +| exception.data.convert-on-type-mismatch | 接收端类型不同时是否转换 | Boolean: true / false | 选填 | true |