Use class reflection to determine TProtocol#109
Open
Yaliang wants to merge 426 commits into
Open
Conversation
upgrade to 0.157
Add new twitter event scriber impl
Prepare for 0.157-tw-0.28 internal release
…onPool add connection pool
…seVersion Update release version to 0.157-tw-0.29
…egrex This is useful when the non-secured port hosted behide a secured front-end service and direct non-secured port access has been limited by a firewall
The authentication over non-secured port is useful when the non-secured port hosted behind a secured front-end service and direct non-secured port access has been limited by the firewall
upgrade to 0.192
Ignore not found files
Release 0.192-tw-0.46
…twitter-forks#136) The built-in host-based service name type for Kerberos authenticator cannot fit our use case. By allowing pass a specific service principle and use the user based name type, the authenticator can be authenticated by Kerberos master without specifying the host running Presto service.
Add utility functions for snowflake ids (https://developer.twitter.com/en/docs/basics/twitter-ids). is_snowflake(id) → boolean Return if a bigint is a snowflake ID (true/false). first_snowflake_for(timestamp) → bigint Return the first snowflake ID given a timestamp. timestamp_from_snowflake(id) → timestamp Return the timestamp given a snowflake ID. cluster_id_from_snowflake(id) → bigint Return the cluster ID given a snowflake ID. instance_id_from_snowflake(id) → bigint Return the instance ID given a snowflake ID. sequence_num_from_snowflake(id) → bigint Return the sequence number given a snowflake ID.
Upgrade to 0.196 based version
Yaliang
force-pushed
the
yaliangw/thrift-generic-row-tprotocal
branch
from
721f286 to
8756d4c
Compare
Yaliang
force-pushed
the
yaliangw/thrift-generic-row-tprotocal
branch
from
8756d4c to
a27435d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Talked with @gerashegalov about the support for varying
TProtocalother thanTBinaryProtocol.Right now, the LZO Thrift read support is working only with TBinaryProtocol or its subclasses(for example, com.twitter.elephantbird.thrift.ThriftBinaryProtocol).
I have double checked that EB currently always pass
TBinaryProtocol's subclassThriftBinaryProtocol.createRecordReaderofcom.twitter.elephantbird.mapreduce.input.MultiInputFormat, if therecordClass(which isThriftGenericRowin our case) is a subclass oforg.apache.thrift.TBaseit will return eitherLzoThriftBlockRecordReaderorLzoThriftB64LineRecordReader.LzoThriftBlockRecordReaderandLzoThriftB64LineRecordReaderwill eventually callcom.twitter.elephantbird.mapreduce.io.ThriftConverter.fromByteswhich usingThriftBinaryDeserializerto wrap binary buffer into aThriftBinaryProtocolThriftBinaryDeserializerwill pass theThriftBinaryProtocolinstance intoThriftGenericRowinstance.This patch would let
ThriftGenericRowbe ready to accept differentTProtocols. However, as long as we are using EB and keepingThriftGenericRowas a subclass ofTBase, we will always receive theThriftBinaryProtocolinstance as the parameter in our LZO Thrift read support implementation.With this patch, if we want to further support other
TProtocol, we can implement differentBinaryConverterProviders for differentTProtocols in EB side and Presto will be ready to accept them without any further change.