Skip to content

Comments

Enable Flink projection pushdown for PSC connector#131

Open
KevBrowne wants to merge 1 commit intopinterest:mainfrom
KevBrowne:enable-flink-projection-pushdown
Open

Enable Flink projection pushdown for PSC connector#131
KevBrowne wants to merge 1 commit intopinterest:mainfrom
KevBrowne:enable-flink-projection-pushdown

Conversation

@KevBrowne
Copy link
Contributor

@KevBrowne KevBrowne commented Feb 23, 2026

Summary

This PR implements SupportsProjectionPushDown for the PSC Flink connector, enabling Flink's optimizer to push column projections down to the source. This optimization reduces deserialization overhead by only deserializing the columns actually needed by queries, rather than deserializing the entire schema.

Key changes:

PscDynamicSource now implements SupportsProjectionPushDown
Added applyProjection() method to compute query-specific projections
Introduced format vs output projection separation (keyFormatProjection/valueFormatProjection for deserialization, keyOutputProjection/valueOutputProjection for row assembly)
Updated getScanRuntimeProvider() and createPscDeserializationSchema() to use the new projections

##Test plan

Unit Test

% mvn -pl psc-flink test -Dtest=PscTableCommonUtilsTest,PscDynamicTableFactoryTest,PscProjectionPushdownTest -Dgpg.skip=true -Djacoco.skip=true
[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.pinterest.psc:psc-examples:jar:4.1.4-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing. @ com.pinterest.psc:psc-java-oss:4.1.4-SNAPSHOT, /home/kbrowne/code/psc/pom.xml, line 155, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.pinterest.psc:psc-integration-test:jar:4.1.4-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing. @ com.pinterest.psc:psc-java-oss:4.1.4-SNAPSHOT, /home/kbrowne/code/psc/pom.xml, line 155, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.pinterest.psc:psc-flink:jar:4.1.4-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.flink:flink-table-planner_${scala.binary.version}:jar -> duplicate declaration of version ${flink.version} @ line 327, column 21
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.flink:flink-json:jar -> duplicate declaration of version ${flink.version} @ line 417, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.pinterest.psc:psc-logging:jar:4.1.4-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing. @ com.pinterest.psc:psc-java-oss:4.1.4-SNAPSHOT, /home/kbrowne/code/psc/pom.xml, line 155, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.pinterest.psc:psc-common:jar:4.1.4-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing. @ com.pinterest.psc:psc-java-oss:4.1.4-SNAPSHOT, /home/kbrowne/code/psc/pom.xml, line 155, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.pinterest.psc:psc-flink-logging:jar:4.1.4-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing. @ com.pinterest.psc:psc-java-oss:4.1.4-SNAPSHOT, /home/kbrowne/code/psc/pom.xml, line 155, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.pinterest.psc:psc-java-oss:pom:4.1.4-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-source-plugin is missing. @ line 155, column 21
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] --------------------< com.pinterest.psc:psc-flink >---------------------
[INFO] Building psc-flink 4.1.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.5:prepare-agent (prepare-unit-tests) @ psc-flink ---
[INFO] Skipping JaCoCo execution because property jacoco.skip is set.
[INFO] argLine set to empty
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ psc-flink ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ psc-flink ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ psc-flink ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 99 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ psc-flink ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 116 source files to /home/kbrowne/code/psc/psc-flink/target/test-classes
[INFO] /home/kbrowne/code/psc/psc-flink/src/test/java/com/pinterest/flink/streaming/connectors/psc/FlinkPscConsumerBaseMigrationTest.java: Some input files use or override a deprecated API.
[INFO] /home/kbrowne/code/psc/psc-flink/src/test/java/com/pinterest/flink/streaming/connectors/psc/FlinkPscConsumerBaseMigrationTest.java: Recompile with -Xlint:deprecation for details.
[INFO] /home/kbrowne/code/psc/psc-flink/src/test/java/com/pinterest/flink/connector/psc/sink/PscSinkITCase.java: /home/kbrowne/code/psc/psc-flink/src/test/java/com/pinterest/flink/connector/psc/sink/PscSinkITCase.java uses or overrides a deprecated API that is marked for removal.
[INFO] /home/kbrowne/code/psc/psc-flink/src/test/java/com/pinterest/flink/connector/psc/sink/PscSinkITCase.java: Recompile with -Xlint:removal for details.
[INFO] /home/kbrowne/code/psc/psc-flink/src/test/java/com/pinterest/flink/connector/psc/dynamic/source/enumerator/StoppablePscEnumContextProxyTest.java: Some input files use unchecked or unsafe operations.
[INFO] /home/kbrowne/code/psc/psc-flink/src/test/java/com/pinterest/flink/connector/psc/dynamic/source/enumerator/StoppablePscEnumContextProxyTest.java: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ psc-flink ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.pinterest.flink.streaming.connectors.psc.table.PscTableCommonUtilsTest
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.638 s - in com.pinterest.flink.streaming.connectors.psc.table.PscTableCommonUtilsTest
[INFO] Running com.pinterest.flink.streaming.connectors.psc.table.PscProjectionPushdownTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.353 s - in com.pinterest.flink.streaming.connectors.psc.table.PscProjectionPushdownTest
[INFO] Running com.pinterest.flink.streaming.connectors.psc.table.PscDynamicTableFactoryTest
[INFO] Tests run: 69, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.12 s - in com.pinterest.flink.streaming.connectors.psc.table.PscDynamicTableFactoryTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 91, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.632 s
[INFO] Finished at: 2026-02-24T14:40:10Z
[INFO] ------------------------------------------------------------------------

Internal Testing E2E w/ Flink

After submitting DDL with 42 columns

SELECT * FROM <Table> LIMIT 10; --> Produced RowType w/ all 42 columns

SELECT a, b, c FROM <Table> LIMIT 10; --> Produced RowType w/ 3 columns a,b,c

SELECT a FROM <table> WHERE b = '<value>' LIMIT 10; Produced Rowtype w/ 2 columns a,b

@KevBrowne KevBrowne force-pushed the enable-flink-projection-pushdown branch from 43f57d1 to 5f25ef7 Compare February 23, 2026 21:41
@KevBrowne KevBrowne marked this pull request as ready for review February 24, 2026 14:34
@KevBrowne KevBrowne requested a review from a team as a code owner February 24, 2026 14:34
@KevBrowne KevBrowne force-pushed the enable-flink-projection-pushdown branch from 5f25ef7 to 813dd37 Compare February 24, 2026 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant