Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] databricks_cluster_library resource #904

Closed
nfx opened this issue Nov 10, 2021 · 0 comments · Fixed by #971
Closed

[FEATURE] databricks_cluster_library resource #904

nfx opened this issue Nov 10, 2021 · 0 comments · Fixed by #971
Labels
feature New feature or request

Comments

@nfx
Copy link
Contributor

nfx commented Nov 10, 2021

Adding libraries to clusters may impose certain challenges mentioned in #820, #468, and #599. These issues become harder, when artifacts are installed outside of terraform workflow.

Scope of the issue is to modify databricks_cluster to ignore any changes to libraries, if they are either managed by new databricks_cluster_library resource or outside of TF workflow.
This was referenced Nov 10, 2021
Closed
Merged
@nfx nfx mentioned this issue Dec 11, 2021
Closed
@nfx nfx added the feature New feature or request label Dec 11, 2021
nfx added a commit that referenced this issue Dec 15, 2021
@nfx
Added databricks_library resource
2b4d2ea 
```hcl
data "databricks_clusters" "all" {
}

resource "databricks_library" "cli" {
  for_each = data.databricks_clusters.all.ids
  cluster_id = each.key
  pypi {
    package = "databricks-cli"
  }
}
```

Fix #904 #954
@nfx nfx mentioned this issue Dec 15, 2021
Merged
nfx added a commit that referenced this issue Dec 15, 2021
@nfx
Added databricks_library resource
8378123 
```hcl
data "databricks_clusters" "all" {
}

resource "databricks_library" "cli" {
  for_each = data.databricks_clusters.all.ids
  cluster_id = each.key
  pypi {
    package = "databricks-cli"
  }
}
```

Fix #904 #954
@nfx nfx closed this as completed in #971 Dec 15, 2021
nfx added a commit that referenced this issue Dec 16, 2021
@nfx
Release v0.4.1
b9c8025 
* Added `databricks_library` resource to install library on `databricks_cluster` ([#904](#904)).
* Added `databricks_clusters` data resource to list all clusters in the workspace, which might be used to install `databricks_library` on all clusters ([#955](#955)).
* Fixed refresh of `library` blocks on a stopped `databricks_cluster` ([#952](#952)).
* Whenever a library fails to get installed on a running `databricks_cluster`, we now automatically remove this library, so that the clean state of managed libraries is properly maintained. Without this fix users had to manually go to Clusters UI and remove library from a cluster, where it failed to install. Libraries add up to CREATE and UPDATE timeouts of `databricks_cluster` resource. ([#599](#599)).
* Added `token` block to `databricks_mws_workspaces` to avoid unnecessary provider aliasing ([#957](#957)).
* Fixed disabling `databricks_global_init_script` ([#958](#958)).
* Fixed configuration drift issues with `aws_attributes`, `azure_attributes`, `gcp_attributes`, and `email_notifications` configuration blocks in `databricks_cluster`, `databricks_job`, and `databricks_instance_pool` resources ([#981](#981)).
* Improved Databricks CLI auth by eagerly resolving `host`, `username`, `password`, and `token` from the specified `profile`. Added explicit logging of auth parameters in debug logs ([#965](#965)).
* TLS timeouts, which may occur during Azure MSI auth, are no longer failing API requests and retried within a normal policy ([#966](#966)).
* `debug_headers` provider conf is also logging the `Host` header to help troubleshooting auth issues ([#964](#964)).
* Added new experimental resources and increased test coverage.

Updated dependency versions:

* Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0
* Bump google.golang.org/api from 0.60.0 to 0.63.0
* Bump github.com/Azure/go-autorest/autorest from 0.11.22 to 0.11.23
* Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.9 to 0.5.10
* Bump gopkg.in/ini.v1 from 1.66.0 to 1.66.2
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.9.0 to 2.10.0
@nfx nfx mentioned this issue Dec 16, 2021
Merged
nfx added a commit that referenced this issue Dec 16, 2021
@nfx
Release v0.4.1 (#983)
da93abe 
Change log:

* Added `databricks_library` resource to install library on `databricks_cluster` ([#904](#904)).
* Added `databricks_clusters` data resource to list all clusters in the workspace, which might be used to install `databricks_library` on all clusters ([#955](#955)).
* Fixed refresh of `library` blocks on a stopped `databricks_cluster` ([#952](#952)).
* Whenever a library fails to get installed on a running `databricks_cluster`, we now automatically remove this library, so that the clean state of managed libraries is properly maintained. Without this fix users had to manually go to Clusters UI and remove library from a cluster, where it failed to install. Libraries add up to CREATE and UPDATE timeouts of `databricks_cluster` resource. ([#599](#599)).
* Added `token` block to `databricks_mws_workspaces` to avoid unnecessary provider aliasing ([#957](#957)).
* Fixed disabling `databricks_global_init_script` ([#958](#958)).
* Fixed configuration drift issues with `aws_attributes`, `azure_attributes`, `gcp_attributes`, and `email_notifications` configuration blocks in `databricks_cluster`, `databricks_job`, and `databricks_instance_pool` resources ([#981](#981)).
* Improved Databricks CLI auth by eagerly resolving `host`, `username`, `password`, and `token` from the specified `profile`. Added explicit logging of auth parameters in debug logs ([#965](#965)).
* TLS timeouts, which may occur during Azure MSI auth, are no longer failing API requests and retried within a normal policy ([#966](#966)).
* `debug_headers` provider conf is also logging the `Host` header to help troubleshooting auth issues ([#964](#964)).
* Added new experimental resources and increased test coverage.

Updated dependency versions:

* Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0
* Bump google.golang.org/api from 0.60.0 to 0.63.0
* Bump github.com/Azure/go-autorest/autorest from 0.11.22 to 0.11.23
* Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.9 to 0.5.10
* Bump gopkg.in/ini.v1 from 1.66.0 to 1.66.2
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.9.0 to 2.10.0
michael-berk pushed a commit to michael-berk/terraform-provider-databricks that referenced this issue Feb 15, 2023
@nfx
Release v0.4.1 (databricks#983)
adf4a40 
Change log:

* Added `databricks_library` resource to install library on `databricks_cluster` ([databricks#904](databricks#904)).
* Added `databricks_clusters` data resource to list all clusters in the workspace, which might be used to install `databricks_library` on all clusters ([databricks#955](databricks#955)).
* Fixed refresh of `library` blocks on a stopped `databricks_cluster` ([databricks#952](databricks#952)).
* Whenever a library fails to get installed on a running `databricks_cluster`, we now automatically remove this library, so that the clean state of managed libraries is properly maintained. Without this fix users had to manually go to Clusters UI and remove library from a cluster, where it failed to install. Libraries add up to CREATE and UPDATE timeouts of `databricks_cluster` resource. ([databricks#599](databricks#599)).
* Added `token` block to `databricks_mws_workspaces` to avoid unnecessary provider aliasing ([databricks#957](databricks#957)).
* Fixed disabling `databricks_global_init_script` ([databricks#958](databricks#958)).
* Fixed configuration drift issues with `aws_attributes`, `azure_attributes`, `gcp_attributes`, and `email_notifications` configuration blocks in `databricks_cluster`, `databricks_job`, and `databricks_instance_pool` resources ([databricks#981](databricks#981)).
* Improved Databricks CLI auth by eagerly resolving `host`, `username`, `password`, and `token` from the specified `profile`. Added explicit logging of auth parameters in debug logs ([databricks#965](databricks#965)).
* TLS timeouts, which may occur during Azure MSI auth, are no longer failing API requests and retried within a normal policy ([databricks#966](databricks#966)).
* `debug_headers` provider conf is also logging the `Host` header to help troubleshooting auth issues ([databricks#964](databricks#964)).
* Added new experimental resources and increased test coverage.

Updated dependency versions:

* Bump github.com/golang-jwt/jwt/v4 from 4.1.0 to 4.2.0
* Bump google.golang.org/api from 0.60.0 to 0.63.0
* Bump github.com/Azure/go-autorest/autorest from 0.11.22 to 0.11.23
* Bump github.com/Azure/go-autorest/autorest/azure/auth from 0.5.9 to 0.5.10
* Bump gopkg.in/ini.v1 from 1.66.0 to 1.66.2
* Bump github.com/hashicorp/terraform-plugin-sdk/v2 from 2.9.0 to 2.10.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Added databricks_library resource databricks/terraform-provider-databricks
1 participant
@nfx