pyspark

#Pre-requisites

1. Python 3.11
2. Pyspark 3.5.1
3. Jupyter 1.1.1

Environment setup

Step1 - Create virtual environment

python -m venv .venv

Step 2 - Create requirement.txt file in project folder and install the same using pip command

Add all the required dependencies

Pyspark package

pyspark==3.5.1

For Notebook usage install jupyter

jupyter==1.1.1

Step 3 - install packages

pip install -r reqruirements.txt

Step 4 - Create a folder ipynb

Step 5 - Create a file test_notebook.ipynb

Step 6 - Choose the correct python interpreter/.venv for the notebook