Development environment for kubernetes, spark-submit and jupyter notebook
Using a kubernetes image build to be ready for Azure Data Lake Storage Gen2 and Delta Lake.
You can create your own spark image from source, the tutorial is inside build-spark-image-from-source
You can choose to do your development on an interactive notebook with jupyter lab, run your script with spark-submit or submit your job to a local Kubernetes.
-
With spark submit
cd spark-submit-env- Use the tutorial inside spark-submit-dev
-
With Jupyter Lab
cd jupyspark-env- Use the tutorial inside jupyspark-env
-
With Kubernetes cluster
cd k8s- Use the tutorial inside k8s