Skip to content

README.md

Latest commit

 

History

History
79 lines (60 loc) · 2.57 KB

README.md

File metadata and controls

79 lines (60 loc) · 2.57 KB
Raw

Scripts

Export parts

The export_parts.py script exports all parts from a MergeTree table to S3/Iceberg. It handles generating ALTER TABLE EXPORT PART commands and runs them in batches and waits until exports in a batch complete before running the next batch. This prevents some parts not exporting due to ClickHouse running out of memory or receiving too many background tasks.

Requirements

On the machine this script will be run:

  • Python 3.10+
  • clickhouse-connect package (pip install clickhouse-connect)

On the ClickHouse server in Altinity Cloud (or elsewhere):

  • Altinity Antalya build of ClickHouse
  • A MergeTree table to export from

Other:

Usage

Basic usage with configuration prompts:

python3 export_parts.py

This will prompt you to enter each configuration value, including ClickHouse connection details, database and table, and S3 details. Values in [brackets] are default values, press enter to accept the default. AWS credentials are optional if you are using the S3 warehouse of the Iceberg catalog in your Altinity Cloud environment.

Before export queries are run, the script will show the first query and how many parts will be exported. In the confirmation prompt type Y to initiate the export.

With config file

You can create a config.json file to load configuration values from. If a config.json file is in the same directory that you execute the script from, it will be used instead of prompting for all values.

Config file format:

{
  "connection": {
    "host": "cluster-name.environment-name.altinity.cloud",
    "port": 8443,
    "secure": true,
    "user": "admin",
    "password": "your-password"
  },
  "export": {
    "database": "ontime",
    "table": "fact_ontime",
    "s3_warehouse": "s3://your-warehouse-bucket",
    "partition_gte": null,
    "partition_lt": null
  },
  "aws_credentials": {
    "access_key_id": "",
    "secret_access_key": "",
    "session_token": ""
  }
}

partition_gte and partition_lt are optional values to filter only specific partitions greater than or equal to partition_gte and less than partition_lt to be exported. This requires the table to have a numeric partition key.

The aws_credentials section is not required for an Altinity Cloud managed Iceberg catalog.

To use a config file not in the current directory, run:

python3 export_parts.py --config path/to/config.json