diff --git a/doc/docs/en/architecture/test_framework.md b/doc/docs/en/architecture/test_framework.md deleted file mode 100644 index 6f5dfd1..0000000 --- a/doc/docs/en/architecture/test_framework.md +++ /dev/null @@ -1 +0,0 @@ -# Test Framework \ No newline at end of file diff --git a/doc/docs/en/getting-started/quick_start.md b/doc/docs/en/getting-started/quick_start.md index ff5ffca..f0150fe 100644 --- a/doc/docs/en/getting-started/quick_start.md +++ b/doc/docs/en/getting-started/quick_start.md @@ -100,7 +100,7 @@ SELECT synchdb_add_conninfo('mysqlconn', 'mysqluser', 'mysqlpwd', 'inventory', - 'postgres', + 'null', 'null', 'null', 'mysql'); @@ -115,7 +115,7 @@ SELECT synchdb_add_conninfo('sqlserverconn', 'sa', 'Password!', 'testDB', - 'postgres', + 'dbo', 'null', 'null', 'sqlserver'); @@ -130,7 +130,7 @@ SELECT synchdb_add_conninfo('oracleconn', 'c##dbzuser', 'dbz', 'FREE', - 'postgres', + 'c##dbzuser', 'null', 'null', 'oracle'); @@ -145,7 +145,7 @@ SELECT synchdb_add_conninfo('ora19cconn', 'DBZUSER', 'dbz', 'FREE', - 'postgres', + 'DBZUSER', 'null', 'null', 'oracle'); @@ -160,7 +160,7 @@ SELECT synchdb_add_conninfo('olrconn', 'DBZUSER', 'dbz', 'FREE', - 'postgres', + 'DBZUSER', 'null', 'null', 'olr'); diff --git a/doc/docs/en/getting-started/remote_database_setups.md b/doc/docs/en/getting-started/remote_database_setups.md index d2155a3..57d22ca 100644 --- a/doc/docs/en/getting-started/remote_database_setups.md +++ b/doc/docs/en/getting-started/remote_database_setups.md @@ -272,9 +272,22 @@ ALTER TABLE products ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ... etc ``` -## **Additional Oracle Setups for Openlog Replicator Support** +## **Set up Native Openlog Replicator Connector for SynchDB** -Openlog Replicator requires additional permissions to stream Oracle changes: +### **Requirements** + +- **Openlog Replicator Version**: `1.3.0` ~ `1.8.5` (verified compatibility for Debezium 2.7.x) +- Oracle instance with redo logs accessible to OLR +- Additional permissions must be granted for OLR (see below) +- Openlog Replicator must be configured and running +- An existing Oracle connector in SynchDB (created using `synchdb_add_conninfo()`) +- <<<**IMPORTANT**>>> **SynchDB must be compiled and built with flag (WITH_OLR=1) to support native openlog replicator connector.** + +Refer to this [external guide](https://highgo.atlassian.net/wiki/external/OTUzY2Q2OWFkNzUzNGVkM2EyZGIyMDE1YzVhMDdkNWE) for details on deploying Openlog Replicator via Docker. + +### **Additional Oracle Permissions** + +In addition to the settings required for Oracle connector, Openlog Replicator requires additional permissions as below: ```sql @@ -318,6 +331,127 @@ BEGIN END; ``` +## **Openlog Replicator Configuration Example** + +SynchDB's OLR support is built against the configuration example below. + +**Version 1.3.0** +```json +{ + "version": "1.3.0", + "source": [ + { + "alias": "SOURCE", + "name": "ORACLE", + "reader": { + "type": "online", + "user": "DBZUSER", + "password": "dbz", + "server": "//ora19c:1521/FREE" + }, + "format": { + "type": "json", + "column": 2, + "db": 3, + "interval-dts": 9, + "interval-ytm": 4, + "message": 2, + "rid": 1, + "schema": 7, + "timestamp-all": 1, + "scn-all": 1 + }, + "memory": { + "min-mb": 64, + "max-mb": 1024 + }, + "filter": { + "table": [ + {"owner": "DBZUSER", "table": ".*"} + ] + }, + "flags": 32 + } + ], + "target": [ + { + "alias": "SYNCHDB", + "source": "SOURCE", + "writer": { + "type": "network", + "uri": "0.0.0.0:7070" + } + } + ] +} + +``` + +**Version 1.8.5** +```json +{ + "version": "1.8.5", + "source": [ + { + "alias": "SOURCE", + "name": "ORACLE", + "reader": { + "type": "online", + "user": "DBZUSER", + "password": "dbz", + "server": "//ora19c:1521/FREE" + }, + "format": { + "type": "json", + "column": 2, + "db": 3, + "interval-dts": 9, + "interval-ytm": 4, + "message": 2, + "rid": 1, + "schema": 7, + "timestamp-all": 1, + "scn-type": 1 + }, + "memory": { + "min-mb": 64, + "max-mb": 1024, + "swap-path": "/opt/OpenLogReplicator/olrswap" + }, + "filter": { + "table": [ + {"owner": "DBZUSER", "table": ".*"} + ] + }, + "flags": 32 + } + ], + "target": [ + { + "alias": "DEBEZIUM", + "source": "SOURCE", + "writer": { + "type": "network", + "uri": "0.0.0.0:7070" + } + } + ] +} + +``` + +Please note the following: + +- "source"."name": "ORACLE" -> this should match the `olr_source` value when defining OLR parameters via `synchdb_add_olr_conninfo()` (See below) +- "source"."reader"."user" -> this should match the `username` value when creating a connector via `synchdb_add_conninfo()` +- "source"."reader"."password" -> this should match the `password` value when creating a connector via `synchdb_add_conninfo()` +- "source"."reader"."server" -> this should contain the values of `hostname`, `port` and `source database` values when creating a connector via `synchdb_add_conninfo()` +- "source"."filter"."table":[] -> this filters the change events that Openlog Replicator captures. <<<**IMPORTANT**>>>: This is currently the only way to filter change events from Oracle as OLR implementations in SynchDB does not do any filtering at this moment. (The `table` and `snapshot table` values are ignored when creating a connector via `synchdb_add_conninfo()`) +- "format":{} -> the specific paylod format ingested by Debezium based or native Openlog Replicator connector. Use these values as specified. +- "memory"."swap-path" -> this tells OLR where to write swap files in low memory scenario. +- "target".[0]."writer"."type": -> this must specify `network` as both Debezium and native Openlog Replicator connector communicate with Openlog Replicator via network +- "target".[0]."writer"."uri": -> this is the bind host and port Openlog Replicator listens on that SynchDB should be able to access via `olr_host` and `olr_port` when defining OLR parameters via `synchdb_add_olr_conninfo()`. + ## **Set up Postgres Connector for SynchDB** PostgreSQL server needs to be configured to be used as a database source to SynchDB diff --git a/doc/docs/en/tutorial/mysql_cdc_to_postgresql.md b/doc/docs/en/tutorial/mysql_cdc_to_postgresql.md index 78b6701..8f9cd74 100644 --- a/doc/docs/en/tutorial/mysql_cdc_to_postgresql.md +++ b/doc/docs/en/tutorial/mysql_cdc_to_postgresql.md @@ -1,4 +1,4 @@ -# MySQL -> PostgreSQL +# MySQL Connector ## **Prepare MySQL Database for SynchDB** @@ -153,7 +153,164 @@ After the initial snapshot, CDC will begin. Restarting a connector in `always` m ## **Preview Source and Destination Table Relationships with schemasync mode** -Before attempting to do an initial snapshot of current table and data, which may be huge, it is possible to "preview" all the tables and data type mappings between source and destination tables before the actual data migration. This gives you an opportunity to modify a data type mapping, or an object name before actual migration happens. This can be done with the special "schemasync" initial snapshot mode. Refer to [object mapping workflow](../../tutorial/object_mapping_workflow/) for a detailed example. +Before attempting to do an initial snapshot of current table and data, which may be huge, it is possible to "preview" all the tables and data type mappings between source and destination tables before the actual data migration. This gives you an opportunity to modify a data type mapping, or an object name before actual migration happens. This can be done with the special "schemasync" initial snapshot mode. + +Please note that you must set `synchdb.olr_snapshot_engine` to 'fdw' in order to use `schemasync` mode to preview the tables. + +### **Create a Connector and Start it in `schemasync` Mode** + +`schemasync` is a special mode that makes the connector connects to remote database and attempt to sync only the schema of designated tables. After this is done, the connector is put to `paused` state and user is able to review all the tables and data types created using the default rules and make change if needed. + +```sql +SELECT synchdb_add_conninfo( + 'mysqlconn', + '127.0.0.1', + 3306, + 'mysqluser', + 'mysqlpwd', + 'inventory', + 'null', + 'null', + 'null', + 'mysql' +); + +SELECT synchdb_start_engine_bgw('mysqlconn', 'schemasync'); +``` + +### **Ensure the connector is put to paused state** + +```sql +SELECT name, connector_type, pid, stage, state FROM synchdb_state_view WHERE name = 'mysqlconn'; + name | connector_type | pid | stage | state +-----------+----------------+---------+---------------------+-------- + mysqlconn | mysql | 1644218 | change data capture | paused + +``` + +### **Review the Tables Created by Default Mapping Rules** + +```sql +SELECT * FROM synchdb_att_view WHERE name = 'mysqlconn'; + name | type | attnum | ext_tbname | pg_tbname | ext_attname | pg_attname | ext_atttypename | pg_atttypename | transform + +-----------+----------+--------+----------------------------+----------------------------+--------------+--------------+-------------------+-------------------+---------------------- +------ + mysqlconn | mysql | 1 | inventory.addresses | inventory.addresses | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.addresses | inventory.addresses | customer_id | customer_id | int | integer | + mysqlconn | mysql | 3 | inventory.addresses | inventory.addresses | street | street | varchar | character varying | + mysqlconn | mysql | 4 | inventory.addresses | inventory.addresses | city | city | varchar | character varying | + mysqlconn | mysql | 5 | inventory.addresses | inventory.addresses | state | state | varchar | character varying | + mysqlconn | mysql | 6 | inventory.addresses | inventory.addresses | zip | zip | varchar | character varying | + mysqlconn | mysql | 7 | inventory.addresses | inventory.addresses | type | type | enum | text | + mysqlconn | mysql | 1 | inventory.customers | inventory.customers | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.customers | inventory.customers | first_name | first_name | varchar | character varying | + mysqlconn | mysql | 3 | inventory.customers | inventory.customers | last_name | last_name | varchar | character varying | + mysqlconn | mysql | 4 | inventory.customers | inventory.customers | email | email | varchar | character varying | + mysqlconn | mysql | 1 | inventory.geom | inventory.geom | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.geom | inventory.geom | g | g | geometry | text | + mysqlconn | mysql | 3 | inventory.geom | inventory.geom | h | h | geometry | text | + mysqlconn | mysql | 1 | inventory.orders | inventory.orders | order_number | order_number | int | integer | + mysqlconn | mysql | 2 | inventory.orders | inventory.orders | order_date | order_date | date | date | + mysqlconn | mysql | 3 | inventory.orders | inventory.orders | purchaser | purchaser | int | integer | + mysqlconn | mysql | 4 | inventory.orders | inventory.orders | quantity | quantity | int | integer | + mysqlconn | mysql | 5 | inventory.orders | inventory.orders | product_id | product_id | int | integer | + mysqlconn | mysql | 1 | inventory.products | inventory.products | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.products | inventory.products | name | name | varchar | character varying | + mysqlconn | mysql | 3 | inventory.products | inventory.products | description | description | varchar | character varying | + mysqlconn | mysql | 4 | inventory.products | inventory.products | weight | weight | float | real | + mysqlconn | mysql | 1 | inventory.products_on_hand | inventory.products_on_hand | product_id | product_id | int | integer | + mysqlconn | mysql | 2 | inventory.products_on_hand | inventory.products_on_hand | quantity | quantity | int | integer | + +``` + +### **Define Custom Mapping (If Needed)** + +```sql +SELECT synchdb_add_objmap('mysqlconn','table','inventory.products','inventory.myproducts'); +SELECT synchdb_add_objmap('mysqlconn','column','inventory.customers.email','contact'); +SELECT synchdb_add_objmap('mysqlconn','datatype','inventory.orders.quantity','bigint|0'); +SELECT synchdb_add_objmap('mysqlconn','transform','inventory.products.name','''>>>>>'' || ''%d'' || ''<<<<<'''); +``` +The above means: + +* source table 'inventory.products' will be mapped to 'inventory.myproducts' in destination +* source column 'inventory.customers.email' will be mapped to 'contact' in destination +* source data type for column 'inventory.orders.quantity' will be mapped to 'bigint' +* source column data 'inventory.products.name' will be transformed accoring to the expression where %d is the data placeholder + +### **Review All Object Mapping Rules Created So Far** + +```sql +SELECT * FROM synchdb_objmap WHERE name = 'mysqlconn'; + name | objtype | enabled | srcobj | dstobj +-----------+-----------+---------+---------------------------+---------------------------- + mysqlconn | table | t | inventory.products | inventory.myproducts + mysqlconn | column | t | inventory.customers.email | contact + mysqlconn | datatype | t | inventory.orders.quantity | bigint|0 + mysqlconn | transform | t | inventory.products.name | '>>>>>' || '%d' || '<<<<<' + +``` + +### **Reload the Object Mapping Rules** + +Once all custom rules have been defined, we need to signal the connector to load them. This will cause the connector to read and apply the object mapping rules. If it sees a discrepancy between current PostgreSQL values and the object mapping values, it will attempt to correct the mapping. + +```sql +SELECT synchdb_reload_objmap('mysqlconn'); + +``` + +### **Review `synchdb_att_view` Again for Changes** + +```sql + name | type | attnum | ext_tbname | pg_tbname | ext_attname | pg_attname | ext_atttypename | pg_atttypename | transform + +-----------+-------+--------+----------------------------+----------------------------+--------------+--------------+-----------------+-------------------+--------------------------- +- + mysqlconn | mysql | 1 | inventory.addresses | inventory.addresses | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.addresses | inventory.addresses | customer_id | customer_id | int | integer | + mysqlconn | mysql | 3 | inventory.addresses | inventory.addresses | street | street | varchar | character varying | + mysqlconn | mysql | 4 | inventory.addresses | inventory.addresses | city | city | varchar | character varying | + mysqlconn | mysql | 5 | inventory.addresses | inventory.addresses | state | state | varchar | character varying | + mysqlconn | mysql | 6 | inventory.addresses | inventory.addresses | zip | zip | varchar | character varying | + mysqlconn | mysql | 7 | inventory.addresses | inventory.addresses | type | type | enum | text | + mysqlconn | mysql | 1 | inventory.customers | inventory.customers | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.customers | inventory.customers | first_name | first_name | varchar | character varying | + mysqlconn | mysql | 3 | inventory.customers | inventory.customers | last_name | last_name | varchar | character varying | + mysqlconn | mysql | 4 | inventory.customers | inventory.customers | email | contact | varchar | character varying | + mysqlconn | mysql | 1 | inventory.geom | inventory.geom | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.geom | inventory.geom | g | g | geometry | text | + mysqlconn | mysql | 3 | inventory.geom | inventory.geom | h | h | geometry | text | + mysqlconn | mysql | 1 | inventory.orders | inventory.orders | order_number | order_number | int | integer | + mysqlconn | mysql | 2 | inventory.orders | inventory.orders | order_date | order_date | date | date | + mysqlconn | mysql | 3 | inventory.orders | inventory.orders | purchaser | purchaser | int | integer | + mysqlconn | mysql | 4 | inventory.orders | inventory.orders | quantity | quantity | int | bigint | + mysqlconn | mysql | 5 | inventory.orders | inventory.orders | product_id | product_id | int | integer | + mysqlconn | mysql | 1 | inventory.products | inventory.myproducts | id | id | int | integer | + mysqlconn | mysql | 2 | inventory.products | inventory.myproducts | name | name | varchar | character varying | '>>>>>' || '%d' || '<<<<<' + mysqlconn | mysql | 3 | inventory.products | inventory.myproducts | description | description | varchar | character varying | + mysqlconn | mysql | 4 | inventory.products | inventory.myproducts | weight | weight | float | real | + mysqlconn | mysql | 1 | inventory.products_on_hand | inventory.products_on_hand | product_id | product_id | int | integer | + mysqlconn | mysql | 2 | inventory.products_on_hand | inventory.products_on_hand | quantity | quantity | int | integer | + + +``` + +### **Resume the Connector or Redo the Entire Snapshot** + +Once the object mappings have been confirmed correct, we can resume the connector. Please note that, resume will proceed to streaming only the new table changes. The existing data of the tables will not be copied. + +```sql +SELECT synchdb_resume_engine('mysqlconn'); +``` + +To capture the table's existing data, we can also redo the entire snapshot with the new object mapping rules. + +```sql +SELECT synchdb_stop_engine_bgw('mysqlconn'); +SELECT synchdb_start_engine_bgw('mysqlconn', 'always'); +``` ## **Selective Table Sync** @@ -241,3 +398,90 @@ postgres=# \dt inventory.* inventory | products | table | ubuntu ``` + +## Secured Connection + +### **Configure Secured Connection** + +to secure the connection to remote database, we need to configure additional SSL related parameters to a connector that has been created by `synchdb_add_conninfo`. The SSL certificates and private keys must be packaged as Java keystore file with a passphrase. These information is then passed to SynchDB via synchdb_add_extra_conninfo(). + +### **synchdb_add_extra_conninfo** + +**Purpose**: Configures extra connector parameters to an existing connector created by `synchdb_add_conninfo` + +| Parameter | Description | Required | Example | Notes | +|:-:|:-|:-:|:-|:-| +| `name` | Unique identifier for this connector | ✓ | `'mysqlconn'` | Must be unique across all connectors | +| `ssl_mode` | SSL mode | ☐ | `'verify_ca'` | can be one of: