SQL Endpoint Permissions - Unable to remove permissions

[jdafoe1]

databricks_permissions resource doesn't remove permissions when applied to SQL Endpoints

Configuration

The following configuration comments out GROUP_TO_REMOVE. On Terarform apply, the Group should be removed from SQL endpoint permissions

resource "databricks_permissions" "sql_endpoint_test" {
  sql_endpoint_id = databricks_sql_endpoint.test.id

  access_control {
    group_name       = "GROUP_TO_KEEP"
    permission_level = "CAN_USE"
  }
  
  # REMOVE THIS GROUP
  #access_control {
  #  group_name       = "GROUP_TO_REMOVE"
  #  permission_level = "CAN_USE"
  #}

}

Terarform apply shows changes successfully applied:

Terraform will perform the following actions:

  # databricks_permissions.sql_endpoint_test will be updated in-place
  ~ resource "databricks_permissions" "sql_endpoint_test" {
        id              = "/sql/endpoints/f9e5be351fb914fb"
        # (2 unchanged attributes hidden)

      - access_control {
          - group_name       = "GROUP_TO_REMOVE" -> null
          - permission_level = "CAN_USE" -> null
        }
      + access_control {
          + group_name       = "GROUP_TO_KEEP"
          + permission_level = "CAN_USE"
        }
      - access_control {
          - group_name       = "GROUP_TO_KEEP" -> null
          - permission_level = "CAN_USE" -> null
        }
    }
	
Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Expected Behaviour

After a Terraform apply, GROUP_TO_REMOVE is removed from permissions on Databricks SQL Endpoint

Actual Behaviour

GROUP_TO_REMOVE doesn't get removed on the Databricks SQL Endpoint. Performing a subsequent plan/apply shows same changes need to be made

Terraform will perform the following actions:

  # databricks_permissions.sql_endpoint_test will be updated in-place
  ~ resource "databricks_permissions" "sql_endpoint_test" {
        id              = "/sql/endpoints/f9e5be351fb914fb"
        # (2 unchanged attributes hidden)

      - access_control {
          - group_name       = "GROUP_TO_REMOVE" -> null
          - permission_level = "CAN_USE" -> null
        }
      + access_control {
          + group_name       = "GROUP_TO_KEEP"
          + permission_level = "CAN_USE"
        }
      - access_control {
          - group_name       = "GROUP_TO_KEEP" -> null
          - permission_level = "CAN_USE" -> null
        }
    }
	
Plan: 0 to add, 1 to change, 0 to destroy.

Steps to reproduce

Apply a databricks_permissions resource on an sql endpoint with an initial set of group permissions. Remove a group permission and run a Terraform apply. The group is not removed.

Terraform and provider versions

Terarform Version 1.0.11
Databricks Provider 0.5.2

Important Factoids

Looking at the DEBUG logs, it seems a PATCH API call is done. Reading the Databricks API docs, shouldn't that be a PUT API call, so that user permisions are overwritten?

2022-03-07T13:22:40.401-0500 [DEBUG] provider.terraform-provider-databricks_v0.5.2: PATCH /api/2.0/permissions/sql/endpoints/f9e5be351fb914fb {
  "access_control_list": [
    {
      "group_name": "GROUP_TO_KEEP",
      "permission_level": "CAN_USE"
    },
    {
      "permission_level": "CAN_MANAGE",
      "user_name": "<redacted>"
    }
  ]
}: timestamp=2022-03-07T13:22:40.401-0500
2022-03-07T13:22:40.801-0500 [DEBUG] provider.terraform-provider-databricks_v0.5.2: 200 OK  {
  "access_control_list": [
    {
      "all_permissions": [
        {
          "inherited": false,
          "permission_level": "CAN_MANAGE"
        }
      ],
      "service_principal_name": "<redacted>"
    },
    {
      "all_permissions": [
        {
          "inherited": false,
          "permission_level": "CAN_USE"
        }
      ],
      "group_name": "GROUP_TO_REMOVE"
    },
    {
      "all_permissions": [
        {
          "inherited": false,
          "permission_level": "CAN_USE"
        }
      ],
      "group_name": "GROUP_TO_KEEP"
    },
      {
      "all_permissions": [
      ... (295 more bytes) <- PATCH /api/2.0/permissions/sql/endpoints/f9e5be351fb914fb: timestamp=2022-03-07T13:22:40.801-0500

PATCH /permissions/sql/endpoints/{endpoint_id}
https://docs.databricks.com/dev-tools/api/latest/permissions.html#operation/set-sqlendpoint-permissions
Grant SQL endpoint permissions for one or more users, groups, or service principals.
This request only grants (adds) permissions. To revoke, use the replace all SQL endpoint permissions operation.

PUT /permissions/sql/endpoints/{endpoint_id}
https://docs.databricks.com/dev-tools/api/latest/permissions.html#operation/update-all-sqlendpoint-permissions
Update all permissions for a specific SQL endpoint, specifying all users, groups or service principal.
WARNING: This request overwrites all existing direct (non-inherited) permissions on the SQL endpoint and replaces it with the new permissions specified in the request body.

[nfx]

https://github.com/databrickslabs/terraform-provider-databricks/blame/master/permissions/resource_permissions.go#L135-L140 we're using PATCH for historical reasons. we'll look into it, given the cycles allowing.