v2.8.2

• Support 0.9.x pymysql in sparkly.testing.MysqlFixture

v2.8.1

• Fix support for using multiple sparkly sessions during tests
• SparklySession does not persist modifications to os.environ
• Support ElasticSearch 7 by making type optional

v2.8.0

• Extend SparklyCatalog to work with database properties

v2.7.1

• Allow newer versions of six package (avoid dependency hell)

v2.7.0

• Migrate to spark 2.4.0
• Fix testing.DataType to use new convention to get field type

v2.6.0

• Add argmax function to sparkly.functions

v2.5.1

• Fix port issue with reading and writing by_url. urlparse return netloc with port, which breaks read and write from MySQL and Cassandra.

v2.5.0

• Add port argument to CassandraFixture and MysqlFixture
• Add Content-Type header to ElasticFixture to support ElasticSearch 6.x
• Update elasticsearch-hadoop connector to 6.5.4
• Update image tag for elasticsearch to 6.5.4

v2.4.1

• Fix write_ext.kafka: run foreachPartition instead of mapPartitions because returned value can cause spark.driver.maxResultSize excess

v2.4.0

• Respect PYSPARK_SUBMIT_ARGS if it is already set by appending SparklySession related options at the end instead of overwriting.
• Fix additional_options to always override SparklySession.options when a session is initialized
• Fix ujson dependency on environments where redis-py is already installed
• Access or initialize SparklySession through get_or_create classmethod
• Ammend sparkly.functions.switch_case to accept a user defined function for deciding whether the switch column matches a specific case