[RFC] Service Credential Binding Rotation for Apps #1198
Conversation
beyhan
requested review from
a team,
rkoster,
stephanme,
ameowlia and
ChrisMcGowan
and removed request for
a team
|
|
||
| ### CF CLI | ||
|
|
||
| The CF CLI should introduce a new flag called `--recreate` for the `bind-service` command. Example: |
There was a problem hiding this comment.
As mentioned during the TOC meeting, recreating in BOSH means deleting an old VM and creating a new one in its place.
There was a problem hiding this comment.
Yes, --recreate is not good.
Maybe:
--create-successor- matches https://github.com/openservicebrokerapi/servicebroker/blob/v2.17/spec.md#request-rotating-a-service-binding--rotate- indicates the use case but is not perfect either because nothing is rotate but only a secondary binding is created--prepare-rotation--duplicate- Any other idea?
There was a problem hiding this comment.
Maybe --clone or --copy? Or maybe no flag is needed and this is just the default behavior (see above).
There was a problem hiding this comment.
I like the idea with no flag and implement duplication/rotation as the default behaviour if a binding already exists. This may result in lots of garbage bindings that are never cleaned up. Could be managed by a limit/quota of service bindings for the same app/service instance.
| - Service binding credentials names remain unique per app but multiple credential bindings to the same service instance can share the same name | ||
|
|
||
| The API `GET /v3/service_credential_bindings` should be extended to return all service credential bindings of an application including the recreated ones. | ||
| When creating the service credential binding information like VCAP_SERVICES or file-based one the CC must select the newest one only. This must be based on the `created_at` field from the credential binding. An application restart or restage as today will be required to activate the new binding. |
There was a problem hiding this comment.
Credhub calls different instances of the same object "versions". Do we want to use similar terminology here?
There was a problem hiding this comment.
Technically, an additional binding to the same service instance is created. It is a real secondary binding with a separate binding guid that just happens to become the successor of the existing binding to the same service instance. Version doesn't fit here. Additional bindings to the same service instance could even have different binding parameters.
"When creating the service credential binding information like VCAP_SERVICES or file-based one the CC must select the newest one only." is a bit un-precise: "... CC must select the newest binding per service instance."
| ``` | ||
| The CLI will use the CF API `GET /v3/service_credential_bindings?app_guids=:guid ` to list the service instance bindings for an application and should delete all old bindings based on the creation date leaving the newest service binding. If no service instance name is provided, the CLI should delete the old bindings of all services currently bound to the application. A full-service binding recreation flow with the CF CLI could look like: | ||
| ``` | ||
| cf bind-service myApp myService -c {<parameters>} --recreate |
There was a problem hiding this comment.
are parameters required on a --recreate? Shoudn't it reuse the previous parameters?
Do we even want to allow different parameters to be supplied on a --recreate?
There was a problem hiding this comment.
With OSBAPI 2.15, the binding parameters are required as CC doesn't store them.
With OSBAPI 2.17, binding parameters are not necessary if the service plan supports binding rotation (see https://github.com/openservicebrokerapi/servicebroker/blob/v2.17/spec.md#binding-rotation).
AFAIK, service binding parameters are rarely used.
--recreate will use the same binding name of the already existing binding. --binding-name parameter is not allowed.
| cf bind-service myApp myService -c {<parameters>} --recreate | ||
| cf bind-service myApp myService2 -c {<parameters>} --recreate | ||
| cf restage myApp --strategy rolling | ||
| cf cleanup-outdated-service-bindings myApp |
There was a problem hiding this comment.
bosh calls unused disks orphaned disks? If we decide to go with versions, this could become cf delete-unused/orphaned-service-binding-versions
There was a problem hiding this comment.
cf delete-orphaned-service-bindings sounds good.
There was a problem hiding this comment.
I wouldn't consider them orphaned, since they're still bound to apps/services.
There was a problem hiding this comment.
We have to find the term which defines this state. In BOSH we use orphaned for disks which are not attached to a VM any more. The old "version" of the binding is also not mapped into the app env and technically not bound to the app. That is why, we considered to use the same term but we can agree also on something different. What would be your suggestion? Maybe inactive and then cf delete-inactive-service-bindings?
|
Some thoughts on binding names:
|
|
|
||
| ## Proposal | ||
|
|
||
| The CC should allow multiple service credential bindings per service instance and application. Only the newest one should be visible in the application VCAP_SERVICES env var (or file-based service binding information) so that existing applications don’t need to change. |
There was a problem hiding this comment.
Could this cause problems for brokers that don't expect multiple bindings for the same app? It's not very clearly specified in the OSBAPI spec.
They get around this in OSBAPI 2.17 by adding the explicit binding_rotatable field to plans.
There was a problem hiding this comment.
I interpreted OSBAPI 2.17 that if binding_rotatable is set, then a service binding can be duplicated by PUT /v2/service_instances/:instance_id/service_bindings/:binding_id with a predecessor_binding_id (Request (Rotating a Service Binding) including binding parameters and any additional state the service broker may have.
I found no reasons why it should not be possible to create a secondary binding for the same app using the regular Request (Creating a Service Binding) independent of binding_rotatable.
If a broker doesn't support multiple bindings for the same app, it should respond with 422 and an error message.
| | parameters | object | A JSON object that is passed to the service broker| | ||
|
|
||
|
|
||
| The result of calling this API is a new service binding with a new guid for the same application and service instance. The initial implementation with the current support of [OSBAPI 2.15](https://github.com/openservicebrokerapi/servicebroker/blob/v2.15/spec.md) in CC can just create a new service binding and with [OSBAPI 2.17](https://github.com/openservicebrokerapi/servicebroker/blob/v2.17/spec.md) adoption in CC it could use the binding rotation support. `POST /v3/service_credential_bindings/:guid/actions/recreate` is just a shortcut for `POST /v3/service_credential_bindings` which would require to specify the app and service instance references which could be used from the specified service binding guid in the URL path. Following requirements should applied to this API: |
There was a problem hiding this comment.
I think a dedicated endpoint for rotating binding credentials was added to OSBAPI, since bindings can be configured with arbitrary parameters, which CC does not persist (and thus cannot be re-issued to the broker when creating the duplicate binding).
It might be confusing for users if:
- At first, calling the
/recreateendpoint doesn't produce an identical binding (due to missing parameters) - Then the behavior changes later when CC adopts OSBAPI 2.17 behind-the-scenes
If we want to have this phased approach, we could instead remove the uniqueness constraint from POST /v3/service_credential_bindings to do the naïve credential duplication and then add dedicated "recreate" behavior when CC adopts OSBAPI 2.17.
There was a problem hiding this comment.
Goal of the RFC was to support service binding credential rotation without waiting for OSBAPI 2.17 support in CC and for all brokers to implement binding_rotatable in their plans. Especially the broker support will probably take ages.
The goal can be achieved without the /recreate endpoint but it puts more complexity on CF API clients like CLI once OSBAPI 2.17 gets available. The client needs to decide if a binding should be rotated in 2.15 or 2.17 style. With /recreate endpoint, this complexity could be handled by the CC.
But yes, we could consider OSBAPI 2.17 as a future problem / rfc.
|
|
||
| ### CF CLI | ||
|
|
||
| The CF CLI should introduce a new flag called `--recreate` for the `bind-service` command. Example: |
There was a problem hiding this comment.
Maybe --clone or --copy? Or maybe no flag is needed and this is just the default behavior (see above).
| ``` | ||
| When the flag `--recreate` is provided the CLI should use the new `POST /v3/service_credential_bindings/:guid/actions/recreate` to recreate the service credential binding. In case there is no service instance binding yet this will fail. | ||
|
|
||
| The cleanup of old inactive service binding should be supported by a new CF CLI command: |
There was a problem hiding this comment.
How will the user know when it is safe to clean up old bindings? How can they audit that running instances are all on the new binding?
There was a problem hiding this comment.
The user has to ensure this by external orchestration of the deployment process. Similar to blue-green deployment. The proposed –strategy rolling-with-binding-rotation in section Future Improvements is such an orchestration.
How can they audit?
- check audit events
cf env <app>and check the binding_guids in VCAP_SERVICEScf ssh <app instance>and check the binding guids in VCAP_SERVICES env var of the process or in file system for file-based service bindings
| cf bind-service myApp myService -c {<parameters>} --recreate | ||
| cf bind-service myApp myService2 -c {<parameters>} --recreate | ||
| cf restage myApp --strategy rolling | ||
| cf cleanup-outdated-service-bindings myApp |
There was a problem hiding this comment.
I wouldn't consider them orphaned, since they're still bound to apps/services.
- CC just allows multiple bindings per app/service - require that binding names don't change on cred rotation - removed --recreate from cf bind-service - added option to keep last x bindings on cleanup
|
Reworked the RFC to address most of the discussion points:
Please review again. |
Link for easy viewing