This guide explains the exact steps a colleague should follow after
cloning the PeMS-Pipeline Git repository to a Windows computer and
opening it in VS Code.
Install the following before starting:
- Git
- Visual Studio Code
- Python 3.12
- Databricks VS Code extension
- Python VS Code extension
- Databricks CLI
Choose a local development directory:
cd C:\devClone the repository:
git clone https://github.com/SANDAG/PeMS-Pipeline.gitEnter the repository:
cd C:\dev\PeMS-PipelineOpen it in VS Code:
code .You should see origin pointing to the PeMS-Pipeline GitHub repository.
Check the current branch:
git branch --show-currentPull the latest version:
git pull origin mainFrom the repository root:
cd C:\dev\PeMS-Pipeline
python3.12 -m venv .venvIf necessary, use:
py -3.12 -m venv .venvActivate the environment:
.\.venv\Scripts\Activate.ps1The terminal should now start with:
(.venv) PS C:\dev\PeMS-Pipeline>
Verify:
python --versionSelect the same interpreter in VS Code:
- Press
Ctrl+Shift+P. - Run
Python: Select Interpreter. - Select:
C:\dev\PeMS-Pipeline\.venv\Scripts\python.exe
The .venv directory is local and should not be committed to Git.
In VS Code, install:
- Databricks
- Python
Open the repository root as the VS Code workspace.
The project should contain files similar to:
PeMS-Pipeline/
├── databricks.yml
├── resources/
│ └── pipeline.yml
└── transformations/
├── 01-bronze_raw_pems.py
├── 02-silver_pems_weekday_counts.py
├── 03-gold_pems_weekday_counts.py
└── holidays_2025.py
databricks.yml defines the Databricks Asset Bundle and target.
resources/pipeline.yml defines the Lakeflow Declarative Pipeline.
transformations/ contains the Python pipeline source files.
If the databricks command is not available, install the current
Databricks CLI:
winget install Databricks.DatabricksCLIClose and reopen VS Code after installation.
Verify:
databricks -vDo not use the legacy Python package pip install databricks-cli for
this bundle workflow.
The workspace URL should be obtained from the team's approved configuration/documentation rather than copied from a personal credential file.
Authenticate:
databricks auth login --host https://<YOUR-DATABRICKS-WORKSPACE-HOST>For example, replace <YOUR-DATABRICKS-WORKSPACE-HOST> with the team's
Azure Databricks workspace hostname.
Complete the browser authentication when prompted.
Verify that the CLI can identify you:
databricks current-user meThe returned userName should be your own Databricks account.
If this returns:
Unauthorized network access to workspace
connect to the required corporate network/VPN and retry. This is a workspace/network-access problem, not a Python pipeline error.
Always validate before deploying:
cd C:\dev\PeMS-Pipeline
databricks bundle validate -t devA successful validation should identify the bundle, target, workspace, and current user.
A warning about /Workspace/Shared being writable by workspace users is
a permissions warning; review it with the team rather than ignoring it
for production deployments.
Optional but useful:
databricks bundle summary -t devThis shows the resources the bundle intends to manage.
The pipeline resource key is the YAML key under:
resources:
pipelines:
pems_pipeline:In this example, the resource key used by CLI commands is:
pems_pipeline
Use the actual key in the repository if it differs.
Deploy the current local bundle:
databricks bundle deploy -t devThis synchronizes the bundle files and creates or updates the Databricks resources defined by the bundle.
Do not manually edit the deployed Python files under
/Workspace/.../files. They are deployment artifacts and may be
overwritten by the next bundle deployment.
If deployment troubleshooting is needed:
databricks bundle deploy -t dev --debugAfter deployment succeeds, run the pipeline resource:
databricks bundle run pems_pipeline -t devIf the repository uses a different resource key, replace pems_pipeline
accordingly.
Do not run individual Lakeflow pipeline source files using
Run File as Workflow when they contain code such as:
from pyspark import pipelines as dpThose files are intended to run as part of the Lakeflow Declarative Pipeline.
Before starting work:
cd C:\dev\PeMS-Pipeline
git checkout main
git pull origin mainEdit the Python/YAML files in VS Code.
Validate:
databricks bundle validate -t devDeploy:
databricks bundle deploy -t devRun:
databricks bundle run pems_pipeline -t devCheck the pipeline result in Databricks.
Check CLI version:
databricks -vCheck authentication:
databricks current-user meValidate the bundle:
databricks bundle validate -t devSummarize bundle resources:
databricks bundle summary -t devSynchronize files:
databricks bundle sync -t devDeploy:
databricks bundle deploy -t devDeploy with debugging output:
databricks bundle deploy -t dev --debugRun the pipeline:
databricks bundle run pems_pipeline -t devList deployed workspace files, adjusting the path to match root_path:
databricks workspace list /Workspace/Shared/PeMS-Pipeline/filesA bundle maintains a relationship between its resource key and an actual Databricks pipeline.
Conceptually:
Bundle resource "pems_pipeline"
|
v
Existing Databricks Pipeline
Use unbind when the bundle should stop managing an existing pipeline
without deleting the pipeline itself:
databricks bundle deployment unbind pems_pipeline -t devAfter unbinding, the old pipeline remains in Databricks. It can still exist, run, and retain ownership of its managed tables.
Do not unbind just because Python code changed. For normal code changes, keep the binding and redeploy:
databricks bundle deploy -t dev
databricks bundle run pems_pipeline -t devConsider unbind when deployment is unexpectedly trying to update,
recreate, or delete an old pipeline that you intentionally want to
preserve.
If an existing Databricks pipeline should become managed by the bundle, bind the bundle resource to its pipeline ID:
databricks bundle deployment bind pems_pipeline <PIPELINE-ID> -t devUse the real Databricks pipeline ID in place of <PIPELINE-ID>.
Binding is useful when migrating a pipeline that was originally created in the Databricks UI and you want to preserve that remote pipeline rather than create another one.
Stopping or unbinding a pipeline does not automatically release ownership of its streaming tables or materialized views.
For example:
Old Pipeline
|
v
travel_data.pems.gold_pems_weekday_counts
If a new pipeline tries to manage the same table, Databricks can reject the run because a managed table can only belong to one pipeline.
For testing a replacement pipeline, use a separate schema or different output table names until the migration is intentionally completed.
The intended workflow is:
GitHub Repository
|
| git pull
v
Local Git Repository in VS Code
|
| edit Python / YAML files
v
Validate Bundle Configuration
|
| databricks bundle validate
| databricks bundle deploy
| databricks bundle run
v
Pipeline Executes on Databricks
|
v
Unity Catalog Tables
|
| verify successful results
v
Commit Changes in VS Code
|
| git push
v
GitHub Repository
GitHub is the source of truth for project code.
VS Code is the local development environment.
The Databricks Bundle deploys the code and resource configuration.
Databricks executes the Lakeflow pipeline.
Unity Catalog stores/manages the resulting tables.
If all developers use the same configuration, for example:
workspace:
root_path: /Workspace/Shared/PeMS-Pipelinethen everyone is deploying to the same bundle location and pipeline resources.
This means one developer's deployment can replace another developer's deployed source code.
Coordinate deployments when sharing a development target.
For a larger team, prefer separate developer-specific dev deployments
and a controlled shared/staging/production target.