[ISSUE] Error when omitting cluster_id when creating a databricks_azure_adls_gen2_mount resource

[fedragon]

Hi Databricks team,

I'm facing an issue when trying to create a databricks_azure_adls_gen2_mount resource without providing a cluster myself (thus asking the provider to create one on its own, as stated here). When I runterraform apply I get a valid plan but trying to apply the changes returns an error saying that AWS attributes can only be provided when using AWS, although I don't set any such attributes in my configuration.

Terraform Version

0.12.26 (I cannot upgrade to the latest at the moment; I have looked through all open/closed issues in this repository but haven't seen my own mentioned there)

Databricks Provider Version

0.2.4

Affected Resource(s)

databricks_azure_adls_gen2_mount

Environment variable names

ARM_CLIENT_ID
ARM_CLIENT_SECRET
ARM_SUBSCRIPTION_ID
ARM_TENANT_ID

Terraform Configuration Files

provider "databricks" {
  azure_client_id      = var.client_id
  azure_client_secret  = var.client_secret
  azure_resource_group = var.resource_group_name
  azure_tenant_id      = data.azurerm_client_config.current.tenant_id
  azure_workspace_name = var.databricks_workspace_name
}

resource "databricks_azure_adls_gen2_mount" "this" {
  storage_account_name   = var.storage_account_name
  container_name         = var.storage_container_name
  mount_name             = var.databricks_mount_name
  tenant_id              = data.azurerm_client_config.current.tenant_id
  client_id              = var.client_id
  client_secret_scope    = databricks_secret_scope.terraform.name
  client_secret_key      = databricks_secret.service_principal.key
  initialize_file_system = true
}

resource "databricks_secret_scope" "terraform" {
  name = "dbx-terraform"
}

resource "databricks_secret" "service_principal" {
  key          = "service_principal_key"
  string_value = var.client_secret
  scope        = databricks_secret_scope.terraform.name
}

Debug Output

N/A: the error only shows after confirming that I want to apply the changes

Panic Output

N/A

Expected Behavior

A small-sized Databricks cluster should have been created and used to mount the filesystem.

Actual Behavior

Terraform returns the following error

Error: aws_attributes can only be specified for clusters running in AWS
Please consult API docs at https://docs.databricks.com/dev-tools/api/latest/clusters.html#create for details

Steps to Reproduce

• terraform apply
• confirm that we want to apply the changes

Important Factoids

None

[nfx]

Hi @fedragon ,

First of all, thanks for report. This issue came as a surprise, because we have TestAzureAccAdlsGen2Mount_correctly_mounts to cover "not specified cluster" case as part of our release procedure.

AWS attributes are added for client not configured with azure auth, which checks for "azure_workspace_resource_id" presence, which may be constructed from subscription, resource group and workspace name. Seems to be a bug with resource id reconstruction, which may touch other things as well.

Can you check if works with azure_workspace_resource_id option, instead of combination of azure_resource_group+azure_subscription_id+azure_workspace_name?

If someone wants to make a PR for this, there needs to be a test case added to TestProviderConfigurationOptions.

[sdebruyn]

I would think it would fetch and set the resource ID at one point so that we can still use that check.

[nfx]

@sdebruyn willing to pick this one up? it's resourceID method + few test cases in TestProviderConfigurationOptions.

[fedragon]

Hi @nfx,

Thanks for the quick reaction! I tried to verify if azure_workspace_resource_id works but have been stopped by the very limited permissions I have on that Azure account (and I don't have access to any other, unfortunately)...

[nfx]

@fedragon i've told few people, someone will pick it up. I'm gathering more contributors to this :)

[sdebruyn]

I've been able to repro it, on it

[sdebruyn]

Looks like I stumbled upon #304 and wasn't actually able to repro this issue.

@fedragon Is the Databricks workspace already created or was it going to be created in the same terraform apply? Could you share the output of terraform plan?

[nfx]

@sdebruyn this issue is unrelated to #304 (az cli), because this one is about client/secret service principal (see env vars). this one is really about things described in #294 (comment)

[fedragon]

@sdebruyn the Databricks workspace already exists, it's not going to be created on the spot

[fedragon]

awesome!