Apache.Arrow.Adbc.Drivers.Databricks 0.20.0

Databricks Driver

The Databricks ADBC driver is built on top of the Spark ADBC driver and inherits all of its properties, plus additional Databricks-specific functionality.

Database and Connection Properties

Note: The Databricks driver inherits all properties from the Spark driver. The properties below are Databricks-specific additions.

Configuration Methods

The Databricks driver supports multiple ways to configure properties:

1. Direct Property Configuration

Pass properties directly when creating the driver connection (traditional method).

2. Environment Variable Configuration

Configure properties using a JSON file loaded via environment variables:

1. Create a JSON configuration file with standard ADBC parameters:

{
  "adbc.databricks.driver_config_take_precedence": "true",
  "adbc.databricks.enable_pk_fk": "false",
  "adbc.connection.catalog": "my_catalog",
  "adbc.connection.db_schema": "my_schema"
}

2. Set the system environment variable DATABRICKS_CONFIG_FILE to point to your JSON file:

   1. Open System Properties → Advanced → Environment Variables
   2. Add new system variable: Name=DATABRICKS_CONFIG_FILE, Value=C:\path\to\your\config.json

3. Property Merging Behavior:

   • By default: Constructor/code properties override environment config properties
   • With "adbc.databricks.driver_config_take_precedence": "true": Environment config properties override constructor/code properties

3. Hybrid Configuration

You can combine both methods - the driver will automatically merge environment config with constructor properties based on the precedence setting.

Use Cases:

• PowerBI Integration: Set system-wide defaults via environment config while allowing connection-specific overrides

Authentication Properties

CloudFetch Properties

CloudFetch is Databricks' high-performance result retrieval system that downloads result data directly from cloud storage.

Databricks-Specific Properties

Tracing Properties

Authentication Methods

The Databricks ADBC driver supports the following authentication methods:

1. Token-based Authentication

Using a Databricks personal access token:

• Set adbc.spark.auth_type to oauth
• Set adbc.databricks.oauth.grant_type to access_token (this is the default if not specified)
• Set adbc.spark.oauth.access_token to your Databricks personal access token

2. OAuth Client Credentials Flow

For machine-to-machine (m2m) authentication:

• Set adbc.spark.auth_type to oauth
• Set adbc.databricks.oauth.grant_type to client_credentials
• Set adbc.databricks.oauth.client_id to your OAuth client ID
• Set adbc.databricks.oauth.client_secret to your OAuth client secret
• Set adbc.databricks.oauth.scope to your auth scope (defaults to sql)

The driver will automatically handle token acquisition, renewal, and authentication with the Databricks service.

Note: Basic (username and password) authentication is not supported at this time.

Server-Side Properties

Server-side properties allow you to configure Databricks session settings. Any property with the adbc.databricks.ssp_ prefix will be passed to the server by executing SET key=value queries.

For example, setting adbc.databricks.ssp_use_cached_result to true will result in executing SET use_cached_result=true on the server when the session is opened.

The property name after the ssp_ prefix becomes the server-side setting name.

Data Types

The following table depicts how the Databricks ADBC driver converts a Databricks type to an Arrow type and a .NET type: