From 9c38de335d18259b79440d1ee57ddf1c172b2a30 Mon Sep 17 00:00:00 2001 From: Johannes Haass Date: Tue, 8 Jul 2025 16:41:08 +0200 Subject: [PATCH 1/4] Create 0014-storage-clis-for-blobstore-operations.md --- ...4-storage-clis-for-blobstore-operations.md | 66 +++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 decisions/0014-storage-clis-for-blobstore-operations.md diff --git a/decisions/0014-storage-clis-for-blobstore-operations.md b/decisions/0014-storage-clis-for-blobstore-operations.md new file mode 100644 index 00000000000..7ace0d7d93e --- /dev/null +++ b/decisions/0014-storage-clis-for-blobstore-operations.md @@ -0,0 +1,66 @@ +# ADR: Introduce Storage CLIs for Blobstore Operations + +## Status + +🔄 **Under Discussion** – This ADR proposes a shared direction for replacing fog-based blobstore implementations. It has not yet been accepted. + +| Provider | Status | Notes | +|----------|--------------------------|---------------------------------------------------------------------------------------------------------| +| Azure | 🚧 PoC in Progress | [PoC](https://github.com/cloudfoundry/cloud_controller_ng/pull/4397) done with `bosh-azure-storage-cli` | +| AWS | 🧭 Open for Contribution | | +| GCP | 🧭 Open for Contribution | | +| Alicloud | 🧭 Open for Contribution | | + + +## Context + +Cloud Controller uses the fog gem family to interface with blobstores like Azure, AWS, GCP, and Alibaba Cloud. +These Ruby gems are largely unmaintained, introducing risks such as: +* Dependency on deprecated SDKs (e.g., Azure SDK for Ruby) +* Blocking Ruby version upgrades +* Potential for unpatched CVEs + +Bosh faces similar issues, as it is also written in Ruby and must interact with blobstores. To address this, BOSH introduced standalone CLI tools (e.g., `bosh-azure-storage-cli`, `bosh-s3cli`) which shell out from Ruby to handle all blobstore operations: +- https://github.com/cloudfoundry/bosh-azure-storage-cli +- https://github.com/cloudfoundry/bosh-s3cli +- https://github.com/cloudfoundry/bosh-gcscli +- https://github.com/cloudfoundry/bosh-ali-storage-cli + +This approach decouples core logic from Ruby gems and has proven to be robust in production. +These CLIs are implemented in Go and use the respective provider SDKs. +All BOSH storage CLIs currently implement a common interface with the following commands: `put`, `get`, `delete`, `exists`, and `sign`. + +A [PoC](https://github.com/cloudfoundry/cloud_controller_ng/pull/4397) has shown that `bosh-azure-storage-cli` can be successfully used in Cloud Controller to push apps. + +## Decision + +Cloud Controller will introduce support for CLI-based blobstore clients, starting with Azure. +Specifically, we will: +* Add a new blobstore client using `bosh-azure-storage-cli` +* Shell out from Cloud Controller to perform blobstore operations +* Allow opt-in via configuration parameter +* Keep the `fog-azure-rm` backend during the transition + +The `bosh-azure-storage-cli` needs to be extended with the following commands: +* `copy` +* `list` +* `properties` +* `ensure-bucket-exists` + +Other providers (AWS, GCP, Alibaba) will follow. Each will require equivalent blobstore clients and support for the above commands. +This will eventually allow us to remove all fog-related gems from Cloud Controller. + +## Consequences + +* Enables removing of `fog-azure-rm` and all other fog related gems +* Reduces long-term maintenance burden and potential security issues +* Allows providers to be migrated independently +* Increases initial complexity during migration phase +* With more consumers, interface changes in the BOSH storage CLIs may require more coordination + +## Alternatives Considered + +* Replace fog with newer Ruby gems → Maintenance risk persists +* Implement own blobstore client in Ruby → High development and testing effort + + From 9feb86e5066223eef300947875a25088d4c38759 Mon Sep 17 00:00:00 2001 From: Johannes Haass Date: Wed, 9 Jul 2025 09:18:43 +0200 Subject: [PATCH 2/4] add examples --- ...4-storage-clis-for-blobstore-operations.md | 83 +++++++++++++++---- 1 file changed, 68 insertions(+), 15 deletions(-) diff --git a/decisions/0014-storage-clis-for-blobstore-operations.md b/decisions/0014-storage-clis-for-blobstore-operations.md index 7ace0d7d93e..1f3788bb4f9 100644 --- a/decisions/0014-storage-clis-for-blobstore-operations.md +++ b/decisions/0014-storage-clis-for-blobstore-operations.md @@ -2,14 +2,14 @@ ## Status -🔄 **Under Discussion** – This ADR proposes a shared direction for replacing fog-based blobstore implementations. It has not yet been accepted. +🔄 **Under Discussion** - This ADR proposes a shared direction for replacing fog-based blobstore implementations. -| Provider | Status | Notes | -|----------|--------------------------|---------------------------------------------------------------------------------------------------------| -| Azure | 🚧 PoC in Progress | [PoC](https://github.com/cloudfoundry/cloud_controller_ng/pull/4397) done with `bosh-azure-storage-cli` | -| AWS | 🧭 Open for Contribution | | -| GCP | 🧭 Open for Contribution | | -| Alicloud | 🧭 Open for Contribution | | +| Provider | Status | Notes | +|--------------|---------------------------|---------------------------------------------------------------------------------------------------------| +| Azure | 🚧 PoC in Progress | [PoC](https://github.com/cloudfoundry/cloud_controller_ng/pull/4397) done with `bosh-azure-storage-cli` | +| AWS | 🧭 Open for Contribution | | +| GCP | 🧭 Open for Contribution | | +| Alibaba Cloud| 🧭 Open for Contribution | | ## Context @@ -20,7 +20,7 @@ These Ruby gems are largely unmaintained, introducing risks such as: * Blocking Ruby version upgrades * Potential for unpatched CVEs -Bosh faces similar issues, as it is also written in Ruby and must interact with blobstores. To address this, BOSH introduced standalone CLI tools (e.g., `bosh-azure-storage-cli`, `bosh-s3cli`) which shell out from Ruby to handle all blobstore operations: +Bosh faces similar issues, as it is also written in Ruby and interacts with blobstores. To address this, Bosh introduced standalone CLI tools which shell out from Ruby to handle all blobstore operations: - https://github.com/cloudfoundry/bosh-azure-storage-cli - https://github.com/cloudfoundry/bosh-s3cli - https://github.com/cloudfoundry/bosh-gcscli @@ -28,13 +28,15 @@ Bosh faces similar issues, as it is also written in Ruby and must interact with This approach decouples core logic from Ruby gems and has proven to be robust in production. These CLIs are implemented in Go and use the respective provider SDKs. -All BOSH storage CLIs currently implement a common interface with the following commands: `put`, `get`, `delete`, `exists`, and `sign`. +All Bosh storage CLIs implement a common interface with the following commands: `put`, `get`, `delete`, `exists`, and `sign`. A [PoC](https://github.com/cloudfoundry/cloud_controller_ng/pull/4397) has shown that `bosh-azure-storage-cli` can be successfully used in Cloud Controller to push apps. +This ADR does not propose breaking changes to existing Bosh storage CLI commands or their output, but outlines necessary additions to support Cloud Controller use cases. It highlights shared concerns and encourages collaboration between Bosh and Cloud Controller. + ## Decision -Cloud Controller will introduce support for CLI-based blobstore clients, starting with Azure. +Cloud Controller will introduce support for CLI based blobstore clients, starting with Azure. Specifically, we will: * Add a new blobstore client using `bosh-azure-storage-cli` * Shell out from Cloud Controller to perform blobstore operations @@ -47,20 +49,71 @@ The `bosh-azure-storage-cli` needs to be extended with the following commands: * `properties` * `ensure-bucket-exists` -Other providers (AWS, GCP, Alibaba) will follow. Each will require equivalent blobstore clients and support for the above commands. -This will eventually allow us to remove all fog-related gems from Cloud Controller. +Other providers (AWS, GCP, Alibaba Cloud) will follow. Each will require equivalent blobstore clients and support for the above commands. +This will eventually allow us to remove all fog related gems from Cloud Controller. ## Consequences -* Enables removing of `fog-azure-rm` and all other fog related gems +* Enables the removal of `fog-azure-rm` and all other fog related gems * Reduces long-term maintenance burden and potential security issues * Allows providers to be migrated independently * Increases initial complexity during migration phase -* With more consumers, interface changes in the BOSH storage CLIs may require more coordination +* With more consumers, interface changes in the Bosh storage CLIs may require more coordination ## Alternatives Considered -* Replace fog with newer Ruby gems → Maintenance risk persists +* Replace fog with newer Ruby gems → Maintenance risk persists and only a short-term solution * Implement own blobstore client in Ruby → High development and testing effort +## Out Of Scope + +* Support for CDNs (currently supported by fog) +* Performance optimizations + +## Example Usage of `bosh-azure-storage-cli` + +### [Bosh](https://github.com/cloudfoundry/bosh/blob/main/src/bosh-director/lib/bosh/director/blobstore/azurestoragecli_blobstore_client.rb) +```Ruby +def object_exists?(object_id) + begin + out, err, status = Open3.capture3(@azure_storage_cli_path.to_s, '-c', @config_file.to_s, 'exists', object_id.to_s) + return true if status.exitstatus.zero? + return false if status.exitstatus == 3 + rescue Exception => e + raise BlobstoreError, e.inspect + end + raise BlobstoreError, "Failed to check existence of az storage account object, code #{status.exitstatus}, output: '#{out}', error: '#{err}'" unless status.success? +end +``` + +### [Cloud Controller PoC](https://github.com/cloudfoundry/cloud_controller_ng/pull/4397) +```Ruby +def exists?(blobstore_key) + key = partitioned_key(blobstore_key) + logger.info("[azure-blobstore] [exists?] Checking existence for: #{key}") + status = run_cli('exists', key, allow_nonzero: true) + + if status.exitstatus == 0 + return true + elsif status.exitstatus == 3 + return false + end + + false +rescue StandardError => e + logger.error("[azure-blobstore] [exists?] azure-storage-cli exists raised error: #{e.message} for #{key}") + false +end + + +def run_cli(command, *args, allow_nonzero: false) + logger.info("[azure-blobstore] Running azure-storage-cli: #{@cli_path} -c #{@config_file} #{command} #{args.join(' ')}") + _, stderr, status = Open3.capture3(@cli_path, '-c', @config_file, command, *args) + return status if allow_nonzero + + raise "azure-storage-cli #{command} failed: #{stderr}" unless status.success? + + status +end +``` \ No newline at end of file From a6704c1c773bca66c348f98da95c2b8cf99c6d25 Mon Sep 17 00:00:00 2001 From: Johannes Haass Date: Wed, 9 Jul 2025 09:39:17 +0200 Subject: [PATCH 3/4] Add Tasks --- ...4-storage-clis-for-blobstore-operations.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/decisions/0014-storage-clis-for-blobstore-operations.md b/decisions/0014-storage-clis-for-blobstore-operations.md index 1f3788bb4f9..da4e05610ae 100644 --- a/decisions/0014-storage-clis-for-blobstore-operations.md +++ b/decisions/0014-storage-clis-for-blobstore-operations.md @@ -52,6 +52,23 @@ The `bosh-azure-storage-cli` needs to be extended with the following commands: Other providers (AWS, GCP, Alibaba Cloud) will follow. Each will require equivalent blobstore clients and support for the above commands. This will eventually allow us to remove all fog related gems from Cloud Controller. +## Tasks + +- [ ] Align with foundational infrastructure working group on proposed CLI usage and extensions +- [ ] Accept this ADR based on shared agreement +- [ ] Extend `bosh-azure-storage-cli` with: + - [ ] `copy` + - [ ] `list` + - [ ] `properties` + - [ ] `ensure-bucket-exists` +- [ ] Implement `bosh-azure-storage-cli` based blobstore client in Cloud Controller with extensibility for other providers in mind +- [ ] Add `bosh-azure-storage-cli` package to capi-release +- [ ] Add support for AWS +- [ ] Add support for GCP +- [ ] Add support for Alibaba Cloud +- [ ] Deprecate/Remove fog once all providers are covered + + ## Consequences * Enables the removal of `fog-azure-rm` and all other fog related gems @@ -116,4 +133,4 @@ def run_cli(command, *args, allow_nonzero: false) status end -``` \ No newline at end of file +``` From 68b17c65557da0b0f6ee4fc337a483ef9581be45 Mon Sep 17 00:00:00 2001 From: Johannes Haass Date: Fri, 11 Jul 2025 13:04:14 +0000 Subject: [PATCH 4/4] add config example --- ...4-storage-clis-for-blobstore-operations.md | 24 +++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/decisions/0014-storage-clis-for-blobstore-operations.md b/decisions/0014-storage-clis-for-blobstore-operations.md index da4e05610ae..f297049de7b 100644 --- a/decisions/0014-storage-clis-for-blobstore-operations.md +++ b/decisions/0014-storage-clis-for-blobstore-operations.md @@ -40,7 +40,22 @@ Cloud Controller will introduce support for CLI based blobstore clients, startin Specifically, we will: * Add a new blobstore client using `bosh-azure-storage-cli` * Shell out from Cloud Controller to perform blobstore operations -* Allow opt-in via configuration parameter +* Allow opt-in via `blobstore_type` configuration parameter and reusing of the existing configuration parameters: + * Example diff: + ```YAML + packages: + app_package_directory_key: app-packages + - blobstore_type: storage-cli + + blobstore_type: fog + fog_connection: + azure_storage_access_key: + azure_storage_account_name: + container_name: app-packages + environment: AzureCloud + provider: AzureRM + max_package_size: 1610612736 + ``` + * Parameters like `fog_connection` may be renamed after the transition period. * Keep the `fog-azure-rm` backend during the transition The `bosh-azure-storage-cli` needs to be extended with the following commands: @@ -75,11 +90,16 @@ This will eventually allow us to remove all fog related gems from Cloud Controll * Reduces long-term maintenance burden and potential security issues * Allows providers to be migrated independently * Increases initial complexity during migration phase +* More maintainers/contributors for the Bosh storage CLIs + + * With more consumers, interface changes in the Bosh storage CLIs may require more coordination ## Alternatives Considered -* Replace fog with newer Ruby gems → Maintenance risk persists and only a short-term solution +* Replace fog with newer Ruby gems + * → Maintenance risk persists and only a short-term solution + * → Not possible for Azure because the used [azure-sdk-for-ruby](https://github.com/Azure/azure-sdk-for-ruby) is archived * Implement own blobstore client in Ruby → High development and testing effort