feat(lance): fix half-float, unsigned int, and struct type mapping#4
Open
jja725 wants to merge 16 commits into
Open
feat(lance): fix half-float, unsigned int, and struct type mapping#4jja725 wants to merge 16 commits into
jja725 wants to merge 16 commits into
Conversation
…ctor Add the protocol translation layer in presto_cpp to convert Java coordinator protocol types to Velox Lance connector types, following the Hive/Iceberg pattern. Changes: - Add datasetPath to Java LanceSplit so native workers can locate datasets - Create C++ protocol types (LanceSplit, LanceTableHandle, LanceColumnHandle, LanceTableLayoutHandle) with JSON serialization matching Java @JsonProperty - Create LancePrestoToVeloxConnector with toVeloxSplit, toVeloxColumnHandle, toVeloxTableHandle conversions - Register Lance connector factory and PrestoToVeloxConnector behind PRESTO_ENABLE_LANCE cmake flag Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire the Lance connector into the same chevron+mustache pipeline that
hive, iceberg, tpch, and arrow_flight use. Hand-written C++ protocol
files are replaced with generated equivalents driven by:
- connector/lance/presto_protocol_lance.yml: 5 abstract subclass
mappings (ColumnHandle, ConnectorTableHandle, ConnectorTableLayoutHandle,
ConnectorSplit, ConnectorTransactionHandle), an UpdateFields entry to
keep LanceTableLayoutHandle::table as shared_ptr<LanceTableHandle>,
and 4 JavaClasses sources.
- connector/lance/presto_protocol-json-{cpp,hpp}.mustache: chevron
templates with the lance namespace; the cpp template hardcodes the
LanceTransactionHandle prolog because the corresponding Java class is
an enum (mirrors the tpch pattern).
- connector/lance/special/LanceColumnHandle.hpp.inc: custom operator<()
override since LanceColumnHandle is comparable.
- connector/lance/special/LanceTransactionHandle.{hpp,cpp}.inc: kept for
parity with tpch's special directory layout.
Makefile gains lance entries in presto_protocol-json, presto_protocol-cpp,
and the .proto target. .gitignore picks up the new intermediate
presto_protocol_lance.json artifact.
Verified end-to-end with `make presto_protocol` — output matches the
prior hand-written types with the same field shapes and signatures, so
LancePrestoToVeloxConnector.cpp continues to compile.
Add a "Lance Connector" subsection under "Build Prestissimo" describing the PRESTO_ENABLE_LANCE cmake flag, what it wires up (LancePrestoToVeloxConnector + velox_lance_connector linkage), and the upstream Velox prerequisite (facebookincubator/velox#16556).
The previous URL (lancedb.github.io/lance) 404s. Point at the upstream repo (github.com/lancedb/lance) — the Lance file format the connector reads, not the LanceDB database product.
The repo moved from the lancedb org to lance-format. Update the link.
…ording The Lance protocol classes are pre-generated and checked in (per the protocol README's "check in these generated files" rule), so saying PRESTO_ENABLE_LANCE "compiles in" the bridge was inaccurate. The flag really only registers the LancePrestoToVeloxConnector and links against velox_lance_connector. Point at the existing protocol README for regeneration instructions. Addresses prestodb#27445 (comment)
Port type-mapping fixes from the Trino Lance connector to Presto SPI: - Map Arrow f16 (Float2Vector) to REAL instead of DOUBLE, widening f16 reads to f32 to meet Presto's REAL minimum (was incorrectly DOUBLE). - Map unsigned int32 (UInt4Vector) to BIGINT and read unsigned int64 (UInt8Vector) without sign loss. - Map Arrow Struct to RowType in LanceColumnHandle so nested struct columns are projected as ROW(...). ArrowBlockBuilder (shared presto-common-arrow) gains Float2/UInt4/UInt8 vector dispatch + assign methods. Bumps lance-core to 4.0.0-beta.13 for the required Arrow vector and dataset APIs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jja725
force-pushed
the
lance-native-connector
branch
from
ca2a2a6 to
829f209
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports type-mapping fixes from the Trino Lance connector to the Presto SPI Lance connector.
Float2Vector) now maps toREAL(was incorrectlyDOUBLE); f16 values are widened to f32 to meet Presto's REAL minimum.UInt4Vector) maps toBIGINT; unsigned int64 (UInt8Vector) is read without sign loss.Structmaps toRowTypeinLanceColumnHandleso nested struct columns project asROW(...).The shared
ArrowBlockBuilder(presto-common-arrow) gainsFloat2/UInt4/UInt8vector dispatch + assign methods. Bumpslance-coreto4.0.0-beta.13for the required Arrow vector / dataset APIs.Test plan
TestLanceArrowBlockBuilder(f16→REAL, unsigned int32/int64 reads), additions toTestLanceColumnHandle(half-float→REAL, unsigned int32→BIGINT, struct→RowType)../mvnw -o test -pl presto-lance→ 24 tests, BUILD SUCCESS.🤖 Generated with Claude Code