Databricks | Insights

🚧

Important information

Third-party documentation is subject to change. Updates might not be reflected in BeyondTrust documentation. For the most up-to-date information, visit the websites linked in this topic.

Prerequisites

• Account Admin role in Databricks account
• Admin role in all Databricks workspaces
• MANAGE access to all workspaces' secret scopes.

Create Databricks connector

1. Select > Insights > Connectors.
2. Select the Available tab.
3. Click Create Connector beside Databricks.
4. Provide a human-readable name for the Databricks connector in the Name field in Step 1.
5. In Step 2, select an installation method:
   1. Databricks CLI (recommended if you can install Databricks CLI)
   2. REST API
6. In Step 3, download the PowerShell script and run it on local machine.
   1. The script will create the service principal in Databricks if it does not exist yet, create secret for service principal (optional), and assign the following permissions to the service principal:
      • Account Admin in Databricks account
      • Admin role in all Databricks workspaces
      • MANAGE scope for all workspaces' secret scopes
   2. Once the script completes, record the output value of Account Id, Client Id and Client Secret for the next step.
   3. If Databricks CLI script option is selected, ensure Databricks CLI is available on the machine (Install or update the Databricks CLI | Databricks Documentation)
7. Use the values recorded from the previous step and enter in Step 4:
   • Account Id: Your Databricks account id, e.g., 123e4567-e89b-12d3-a456-426614174111.
   • Client Id: Databricks service principal application Id.
   • Client Secret: Databricks service principal secret value.
8. Click Create Connector.

Create Databricks Service Principal

📘

The following instructions for the Databricks console provide an alternative method for executing the setup script. If the setup script completes successfully, there is no need to perform the steps outlined here.

1. Create Service Principal
   1. Click User management on the sidebar.
   2. On Service Principals tab, click Add service principal.
   3. Enter BT-Insights-Databricks-SP service principal’s name.
   4. Click Add service principal.

ℹ️

For more information, see Manage service principals | Databricks Documentation.

2. Create Credentials for Service Principal
   1. In the sidebar, click User management.
   2. On the Service principals tab, find and click the username.
   3. Select Credentials & secret tab.
   4. Click Generate secret.
   5. Enter the secret lifetime, e.g.: 365 and click Generate.
   6. Record the Client Id and Secret.

Assign Databricks Service Principal Permission

Assign Account Admin role to Service Principal

1. In the sidebar, click User management.
2. On the Service principals tab, find and click the username.
3. Select Roles tab.
4. Select Account admin to activate.

ℹ️

For more information, see Manage service principals | Databricks Documentation.

Assign Admin role to service principals for all workspaces

1. On the menu on the left, navigate to Workspaces.
2. Do the following step for each workspace:
   1. Click on workspace name.
   2. Navigate to Permissions tab.
   3. Click Add permissions.
   4. Search for BT-Insights-Databricks-SP.
   5. Select Admin for Permission.
   6. Click Save.

Assign MANAGE Scope to all workspaces' secret scopes

1. Ensure Databricks CLI is installed. For more information, see Databricks CLI documentation.
2. Obtain the Databricks Account ID by clicking on the user profile located in the top right corner of the Databricks account portal.
3. Log in to Databricks account:

   databricks auth login --host <https://accounts.cloud.databricks.com> --account-id {AccountId}
   

4. List all workspaces:

   databricks account workspaces list --output json
   

5. For each workspace that is not in FAILED state:
   1. Get workspace deployment_name from the output above
   2. Log in to each workspace

      databricks auth login --host https\://{workspace_deployment_name}.cloud.databricks.com --profile {workspace_deployment_name}
      

   3. List all secret scopes

      databricks secrets list-scopes --profile {workspace_deployment_name}
      

   4. For each secret scope
      1. Assign secret scope MANAGE permission to service principal

         databricks secrets put-acl {scope_name} {sp_client_id} MANAGE --profile {workspace_deployment_name}