-
Notifications
You must be signed in to change notification settings - Fork 0
CXH-2166: implement PAT (workspace token) authentication #54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
c4270b3
5d69329
850b2a6
c08d8b1
870c90c
8b12e6a
df53744
27f217d
c29d543
423d12e
c6917da
622422d
b7158d4
f1a5ff0
4527918
b691ba6
b06c054
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,20 +18,18 @@ right top corner that will open a dropdown menu with the account ID along other | |
| options. | ||
|
|
||
| Another requirement is to have valid credentials to run the connector with. This | ||
| will decide how connector will be executed. You can use either OAuth client | ||
| credentials flow or Basic auth flow (username and password) or Bearer auth flow. | ||
| Both OAuth and Basic can be used across account and all workspaces you have | ||
| access to. Bearer auth can be used only for a specific workspace. | ||
| will decide how connector will be executed. You can use either the OAuth client | ||
| credentials flow or the Bearer auth flow. OAuth can be used across account and | ||
| all workspaces you have access to. Bearer auth can be used only for a specific | ||
| workspace. | ||
|
|
||
| To use the OAuth, you need to create a service principal and add OAuth secret | ||
| (client id and secret) to it. You can do that by going to the user management | ||
| tab and clicking on the Service Principals tab. Then click on the Add Service | ||
| principal button and name it. You then need to add OAuth secret to it by | ||
| clicking on the Generate secret button. You can use this secret to authenticate | ||
| across all workspaces that service principal has access to. To use basic auth, | ||
| you just need to provide a username and password of a user that has access to | ||
| the Databricks API. Both methods require admin access to the Databricks account | ||
| and each workspace you want to sync. | ||
| across all workspaces that service principal has access to. This requires admin | ||
| access to the Databricks account and each workspace you want to sync. | ||
|
|
||
| To use bearer auth, you need to provide a Databricks workspace access token. You | ||
| can create a new token by logging into the workspace and going into user | ||
|
|
@@ -55,14 +53,14 @@ baton-databricks --hostname "azuredatabricks.net" | |
| ``` | ||
| brew install conductorone/baton/baton conductorone/baton/baton-databricks | ||
|
|
||
| BATON_ACCOUNT_ID=account_id BATON_USERNAME=username BATON_PASSWORD=password baton-databricks | ||
| BATON_ACCOUNT_ID=account_id BATON_DATABRICKS_CLIENT_ID=client_id BATON_DATABRICKS_CLIENT_SECRET=client_secret baton-databricks | ||
| baton resources | ||
| ``` | ||
|
|
||
| ## docker | ||
|
|
||
| ``` | ||
| docker run --rm -v $(pwd):/out -e BATON_ACCOUNT_ID=account_id BATON_USERNAME=username BATON_PASSWORD=password ghcr.io/conductorone/baton-databricks:latest -f "/out/sync.c1z" | ||
| docker run --rm -v $(pwd):/out -e BATON_ACCOUNT_ID=account_id BATON_DATABRICKS_CLIENT_ID=client_id BATON_DATABRICKS_CLIENT_SECRET=client_secret ghcr.io/conductorone/baton-databricks:latest -f "/out/sync.c1z" | ||
| docker run --rm -v $(pwd):/out ghcr.io/conductorone/baton:latest -f "/out/sync.c1z" resources | ||
| ``` | ||
|
|
||
|
|
@@ -72,7 +70,7 @@ docker run --rm -v $(pwd):/out ghcr.io/conductorone/baton:latest -f "/out/sync.c | |
| go install github.com/conductorone/baton/cmd/baton@main | ||
| go install github.com/conductorone/baton-databricks/cmd/baton-databricks@main | ||
|
|
||
| BATON_ACCOUNT_ID=account_id BATON_USERNAME=username BATON_PASSWORD=password baton-databricks | ||
| BATON_ACCOUNT_ID=account_id BATON_DATABRICKS_CLIENT_ID=client_id BATON_DATABRICKS_CLIENT_SECRET=client_secret baton-databricks | ||
| baton resources | ||
| ``` | ||
|
|
||
|
|
@@ -97,6 +95,11 @@ both flags at the same time. If you do that, connector will sync with all | |
| workspaces that are associated with provided tokens and all workspaces that are | ||
| in the list of workspaces. | ||
|
|
||
| When authenticating with `--workspace-tokens` instead of the OAuth client ID and | ||
| secret, also pass `--auth-method workspace-token` (or set | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: while cleaning up these auth docs, the paragraph just above (lines 88-96) is now stale. |
||
| `BATON_AUTH_METHOD=workspace-token`), otherwise the connector validates against | ||
| the OAuth fields by default and rejects the config. | ||
|
|
||
| To instead exclude specific workspaces from the sync, pass them to the | ||
| `--databricks-exclude-workspaces` flag (or the | ||
| `BATON_DATABRICKS_EXCLUDE_WORKSPACES` environment variable) as a comma-separated | ||
|
|
@@ -129,25 +132,47 @@ Usage: | |
| Available Commands: | ||
| capabilities Get connector capabilities | ||
| completion Generate the autocompletion script for the specified shell | ||
| config Get the connector config schema | ||
| health-check Check the health of a running connector | ||
| help Help about any command | ||
|
|
||
| Flags: | ||
| --account-hostname string The hostname used to connect to the Databricks account API. If not set, it will be calculated from the hostname field. ($BATON_ACCOUNT_HOSTNAME) | ||
| --account-id string required: The Databricks account ID used to connect to the Databricks Account and Workspace API ($BATON_ACCOUNT_ID) | ||
| --client-id string The client ID used to authenticate with ConductorOne ($BATON_CLIENT_ID) | ||
| --client-secret string The client secret used to authenticate with ConductorOne ($BATON_CLIENT_SECRET) | ||
| --databricks-client-id string The Databricks service principal's client ID used to connect to the Databricks Account and Workspace API ($BATON_DATABRICKS_CLIENT_ID) | ||
| --databricks-client-secret string The Databricks service principal's client secret used to connect to the Databricks Account and Workspace API ($BATON_DATABRICKS_CLIENT_SECRET) | ||
| --databricks-exclude-workspaces strings Workspaces to exclude from sync, identified by workspace name, deployment name, or numeric workspace ID ($BATON_DATABRICKS_EXCLUDE_WORKSPACES) | ||
| -f, --file string The path to the c1z file to sync with ($BATON_FILE) (default "sync.c1z") | ||
| -h, --help help for baton-databricks | ||
| --hostname string The Databricks hostname used to connect to the Databricks API ($BATON_HOSTNAME) (default "cloud.databricks.com") | ||
| --log-format string The output format for logs: json, console ($BATON_LOG_FORMAT) (default "json") | ||
| --log-level string The log level: debug, info, warn, error ($BATON_LOG_LEVEL) (default "info") | ||
| -p, --provisioning This must be set in order for provisioning actions to be enabled ($BATON_PROVISIONING) | ||
| --skip-full-sync This must be set to skip a full sync ($BATON_SKIP_FULL_SYNC) | ||
| --ticketing This must be set to enable ticketing support ($BATON_TICKETING) | ||
| -v, --version version for baton-databricks | ||
| --account-hostname string The hostname used to connect to the Databricks account API. If not set, it will be calculated from the hostname field. ($BATON_ACCOUNT_HOSTNAME) | ||
| --account-id string required: The Databricks account ID used to connect to the Databricks Account and Workspace API ($BATON_ACCOUNT_ID) | ||
| --auth-method string ($BATON_AUTH_METHOD) | ||
| --client-id string The client ID used to authenticate with ConductorOne ($BATON_CLIENT_ID) | ||
| --client-secret string The client secret used to authenticate with ConductorOne ($BATON_CLIENT_SECRET) | ||
| --databricks-client-id string required: The Databricks service principal's client ID used to connect to the Databricks Account and Workspace API ($BATON_DATABRICKS_CLIENT_ID) | ||
| --databricks-client-secret string required: The Databricks service principal's client secret used to connect to the Databricks Account and Workspace API ($BATON_DATABRICKS_CLIENT_SECRET) | ||
| --databricks-exclude-workspaces strings Workspaces to exclude from sync, identified by workspace name, deployment name, or numeric workspace ID ($BATON_DATABRICKS_EXCLUDE_WORKSPACES) | ||
| --external-resource-c1z string The path to the c1z file to sync external baton resources with ($BATON_EXTERNAL_RESOURCE_C1Z) | ||
| --external-resource-entitlement-id-filter string The entitlement that external users, groups must have access to sync external baton resources ($BATON_EXTERNAL_RESOURCE_ENTITLEMENT_ID_FILTER) | ||
| --external-resource-traits strings Resource type traits (e.g. "user", "group", "app") to sync and match from the external resource c1z. When unset the matcher falls back to user and group; passing this flag replaces the full set rather than adding to it. ($BATON_EXTERNAL_RESOURCE_TRAITS) | ||
| -f, --file string The path to the c1z file to sync with ($BATON_FILE) (default "sync.c1z") | ||
| --health-check Enable the HTTP health check endpoint ($BATON_HEALTH_CHECK) | ||
| --health-check-port int Port for the HTTP health check endpoint ($BATON_HEALTH_CHECK_PORT) (default 8081) | ||
| -h, --help help for baton-databricks | ||
| --hostname string The Databricks hostname used to connect to the Databricks API ($BATON_HOSTNAME) (default "cloud.databricks.com") | ||
| --http-timeout-seconds int HTTP client timeout in seconds (max 1800) ($BATON_HTTP_TIMEOUT_SECONDS) (default 300) | ||
| --keep-previous-sync-c1z Keep the previously synced c1z on disk to enable ETag replay across service-mode syncs (requires a connector that supports ETag replay; costs one c1z of local disk) ($BATON_KEEP_PREVIOUS_SYNC_C1Z) | ||
| --log-format string The output format for logs: json, console ($BATON_LOG_FORMAT) (default "json") | ||
| --log-level string The log level: debug, info, warn, error ($BATON_LOG_LEVEL) (default "info") | ||
| --log-level-debug-expires-at string The timestamp indicating when debug-level logging should expire ($BATON_LOG_LEVEL_DEBUG_EXPIRES_AT) | ||
| --log-path strings The file path to write logs to ($BATON_LOG_PATH) | ||
| --otel-collector-endpoint string The endpoint of the OpenTelemetry collector to send observability data to (used for both tracing and logging if specific endpoints are not provided) ($BATON_OTEL_COLLECTOR_ENDPOINT) | ||
| --parallel-sync Deprecated: use --workers instead. ($BATON_PARALLEL_SYNC) | ||
| -p, --provisioning This must be set in order for provisioning actions to be enabled ($BATON_PROVISIONING) | ||
| --skip-entitlements-and-grants This must be set to skip syncing of entitlements and grants ($BATON_SKIP_ENTITLEMENTS_AND_GRANTS) | ||
| --skip-full-sync This must be set to skip a full sync ($BATON_SKIP_FULL_SYNC) | ||
| --storage-engine string The storage engine to use when opening the sync c1z file: sqlite or pebble. Leave unset to use the baton-sdk default. ($BATON_STORAGE_ENGINE) | ||
| --sync-resource-types strings The resource type IDs to sync ($BATON_SYNC_RESOURCE_TYPES) | ||
| --sync-resources strings The resource IDs to sync ($BATON_SYNC_RESOURCES) | ||
| --task-concurrency int The number of Baton tasks to run concurrently in service mode. Tasks may include sync, grant, revoke, and more. Minimum value is 1, maximum value is 100. ($BATON_TASK_CONCURRENCY) (default 3) | ||
| --ticketing This must be set to enable ticketing support ($BATON_TICKETING) | ||
| -v, --version version for baton-databricks | ||
| --workers int The number of sync workers to use. -1 for auto-detect, 0 for sequential, >0 for parallel ($BATON_WORKERS) | ||
| --workspace-tokens strings required: The Databricks personal access tokens scoped to specific workspaces used to connect to the Databricks Workspace API ($BATON_WORKSPACE_TOKENS) | ||
| --workspaces strings Limit syncing to the specified workspaces, by deployment name, not workspace ID. Required when using workspace tokens, in the same order as workspace-tokens. ($BATON_WORKSPACES) | ||
|
|
||
| Use "baton-databricks [command] --help" for more information about a command. | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -140,14 +140,82 @@ | |
| "defaultValue": "cloud.databricks.com" | ||
| } | ||
| }, | ||
| { | ||
| "name": "workspaces", | ||
| "displayName": "Workspaces", | ||
| "description": "Limit syncing to the specified workspaces, by deployment name, not workspace ID. Required when using workspace tokens, in the same order as workspace-tokens. Mutually exclusive with databricks-exclude-workspaces.", | ||
| "stringSliceField": {} | ||
| }, | ||
| { | ||
| "name": "workspace-tokens", | ||
| "displayName": "Workspace Tokens", | ||
| "description": "The Databricks personal access tokens scoped to specific workspaces used to connect to the Databricks Workspace API", | ||
| "isRequired": true, | ||
| "isSecret": true, | ||
| "stringSliceField": { | ||
| "rules": { | ||
| "isRequired": true | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "name": "databricks-exclude-workspaces", | ||
| "displayName": "Exclude Workspaces", | ||
| "description": "Workspaces to exclude from sync, identified by workspace name, deployment name, or numeric workspace ID", | ||
| "description": "Workspaces to exclude from sync, identified by workspace name, deployment name, or numeric workspace ID. Mutually exclusive with workspaces.", | ||
| "stringSliceField": {} | ||
| } | ||
| ], | ||
| "constraints": [ | ||
| { | ||
| "kind": "CONSTRAINT_KIND_MUTUALLY_EXCLUSIVE", | ||
| "fieldNames": [ | ||
| "workspaces", | ||
| "databricks-exclude-workspaces" | ||
| ] | ||
| }, | ||
| { | ||
| "kind": "CONSTRAINT_KIND_DEPENDENT_ON", | ||
| "fieldNames": [ | ||
| "workspace-tokens" | ||
| ], | ||
| "secondaryFieldNames": [ | ||
| "workspaces" | ||
| ] | ||
| } | ||
| ], | ||
| "displayName": "Databricks", | ||
| "helpUrl": "/docs/baton/databricks", | ||
| "iconUrl": "/static/app-icons/databricks.svg" | ||
| "iconUrl": "/static/app-icons/databricks.svg", | ||
| "fieldGroups": [ | ||
| { | ||
| "name": "oauth2", | ||
| "displayName": "OAuth2", | ||
| "helpText": "Authenticate as a service principal using an OAuth2 client ID and secret.", | ||
| "fields": [ | ||
| "account-id", | ||
| "databricks-client-id", | ||
| "databricks-client-secret", | ||
| "hostname", | ||
| "account-hostname", | ||
| "workspaces", | ||
| "base-url", | ||
| "databricks-exclude-workspaces" | ||
| ], | ||
| "default": true | ||
| }, | ||
| { | ||
| "name": "workspace-token", | ||
| "displayName": "Workspace token", | ||
| "helpText": "Authenticate with a personal access token scoped to each workspace.", | ||
| "fields": [ | ||
| "account-id", | ||
| "workspaces", | ||
| "workspace-tokens", | ||
| "hostname", | ||
| "account-hostname", | ||
| "base-url", | ||
| "databricks-exclude-workspaces" | ||
| ] | ||
| } | ||
| ] | ||
|
Comment on lines
+189
to
+220
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: this generated artifact is stale relative to 🤖 via Claude Code |
||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -42,7 +42,7 @@ A user with the **Account admin** role in each Databricks workspace you want to | |
|
|
||
| ### Generate Databricks credentials | ||
|
|
||
| You have three authentication choices when setting up the Databricks connector: | ||
| You have two authentication choices when setting up the Databricks connector: | ||
|
|
||
| - **OAuth** (syncs info from all Databricks workspaces) | ||
|
|
||
|
|
@@ -69,10 +69,6 @@ You have three authentication choices when setting up the Databricks connector: | |
| </Step> | ||
| </Steps> | ||
|
|
||
| - **Username and password** (syncs info from all Databricks workspaces) | ||
|
|
||
| You do not need to generate any additional credentials to use this method. | ||
|
|
||
| **Done.** Here's the set of credentials you'll need when setting up the connector: | ||
|
|
||
| - Account ID | ||
|
|
@@ -83,13 +79,7 @@ OR | |
|
|
||
| - Account ID | ||
| - Personal access token | ||
| - Workspace ID for the Databricks workspace you're syncing | ||
|
|
||
| OR | ||
|
|
||
| - Account ID | ||
| - Username | ||
| - Password | ||
| - Deployment name of the Databricks workspace you're syncing (the subdomain in the workspace URL, not the workspace ID) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Docs] Same "Username and password" gap as README.md Same note as README.md: this page's auth-choices list (and the Kubernetes secret example further down) also still promises a "Username and password" auth method with
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. same, pulled the username/password bits from |
||
|
|
||
| Next, move on to the instructions for your chosen setup method. | ||
|
|
||
|
|
@@ -132,13 +122,13 @@ To complete this task, you'll need: | |
| Find the **Settings** area of the page and click **Edit**. | ||
| </Step> | ||
| <Step> | ||
| Select whether you're authenticating with **OAuth**, a **Personal access token**, or your **Username and password**. | ||
| Select whether you're authenticating with **OAuth** or a **Personal access token**. | ||
| </Step> | ||
| <Step> | ||
| Paste the account ID you looked up in Step 1 into the **Account ID** field. | ||
| </Step> | ||
| <Step> | ||
| Enter the required OAuth, token, or username and password credentials into the other two fields. | ||
| Enter the required OAuth or token credentials into the other two fields. | ||
| </Step> | ||
| <Step> | ||
| **Google Cloud Platform and Azure Databricks customers only:** Enter your Databricks account hostname and hostname in the relevant fields. | ||
|
|
@@ -226,13 +216,9 @@ stringData: | |
|
|
||
| # Databricks credentials, option 2 | ||
| BATON_ACCOUNT_ID: <Databricks account ID> | ||
| BATON_AUTH_METHOD: workspace-token | ||
| BATON_WORKSPACE_TOKENS: <Personal access token> | ||
| BATON_WORKSPACES: <Workspace ID for the Databricks workspace you're syncing> | ||
|
|
||
| # Databricks credentials, option 3 | ||
| BATON_ACCOUNT_ID: <Databricks account ID> | ||
| BATON_USERNAME: <Username for the Databricks account> | ||
| BATON_PASSWORD: <Password for the Databricks account> | ||
| BATON_WORKSPACES: <Deployment name of the Databricks workspace you're syncing, not the workspace ID> | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: Option 2 needs |
||
|
|
||
| # Optional: comma-separated workspaces to exclude from sync (workspace name, deployment name, or numeric ID) | ||
| BATON_DATABRICKS_EXCLUDE_WORKSPACES: <workspace-a,workspace-b> | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Suggestion: this commit adds |
||
|
|
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Docs] Separate from this PR — "Username and password" auth is still documented but doesn't exist
Not something this PR needs to fix, but since you're already cleaning up the auth docs in this exact file: further up (README.md's "Bearer auth"/basic-auth paragraphs, around the intro section) this README still describes a third auth option — username/password ("Basic auth") — and there's genuinely zero implementation of it anywhere in the codebase (no
BasicAuthtype inpkg/databricks/auth.go, no username/password fields inpkg/config/config.go). Might be worth a follow-up to either implement it or drop the claim, since a customer following these docs for that option will configure something that silently doesn't exist.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pulled the username/password bits from the README in this PR. databricks EOL'd basic auth back in 2024 and it was never implemented here, so the docs were just stale.
https://docs.databricks.com/aws/en/security/auth/password-deprecation