Serverless updates needed #202
Description
Currently need to run through the following to make this workable with serverless:
- Remove compute/cluster details from resource .yml files (for serverless execution)
- Manually set the workspace Environment Verison to 2 in each notebook that is configured to run as a job. This cannot be configured within DAB yaml for Notebook tasks and resolves package import errors
- Remove ‘requirements.txt’ install to avoid unnecessary dependencies
- Remove ‘dbutils.library.restartPython()’ cells
- Resolves compute connection issues
Suggest making appropriate updates to the repo so serverless mlops-stacks is more turnkey. Also update the dependencies in MLOps Stacks to the latest versions (like MLflow 3). Also consider changing the jobs by default to serverless.