Conversation
- Added support for all four merge strategies (insert, update, update_insert, delete_insert) in D1, Databricks, Exasol, Oracle, Postgres, Snowflake, SQL Server, and SQLite templates. - Implemented specific merge strategies for DuckDB, MariaDB, MySQL, Redshift, and StarRocks, noting limitations where applicable. - Updated task_run_write.go to allow dynamic selection of merge strategies based on configuration.
…ring - enhance template evaluator to recursively find template expressions within nested braces - enables rendering of complex structures like graphql queries where outer braces are not expressions - introduce `evaluatorSafe` in `APIConnection` to keep missing expressions intact during rendering - add `renderStringSafe` and `renderAnySafe` methods to `APIConnection`, `Endpoint`, and `Iteration` - add comprehensive test cases for nested expression parsing and safe rendering behavior
- Added foreign key DDL templates and column/table comment templates for Oracle, PostgreSQL, Redshift, Snowflake, and SQL Server. - Implemented extended column attributes and foreign key relationships queries for Oracle, PostgreSQL, Redshift, Snowflake, and SQL Server. - Introduced default value translation mappings for schema migration in Oracle, PostgreSQL, Redshift, Snowflake, and SQLite. - Updated SQLite template to handle foreign key relationships and extended column attributes. - Refactored the ReplicationConfig to include schema migration capabilities, including ordering streams by foreign key dependencies. - Enhanced task functions to apply schema attributes (foreign keys, indexes) during data writes. - Adjusted test script timeout for CLI tests.
…using INFORMATION_SCHEMA
… metadata query to exclude PRIMARY key
…kDB, and ClickHouse
…SON SQLite test refactor(datastream): clean up SetConfig method by removing unused column assignment
…n table filtering
- introduce `jq` as a source option for JSON-based data streams - enable `jq` for API source configuration and data extraction - implement `jq` processing within `JSONStream` for various database sources - add `JqCompile` and `JqRun` utilities with caching for efficient `jq` expression execution - include comprehensive test cases for `jq` source option and `jq()` transform function
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sling v1.5.7
Schema Migration
Merge Strategies
insert,update,update_insert, anddelete_insertmerge strategiesjqExpressionsjqsource option for JSON-based data streamsjq()transform function for column-level transformationsTemplate & Expression Engine
Fixes & Improvements
Discovermethodcurrent_database()for catalog filtering