Persistent Volumes being created twice

[ryaneorth]

/kind bug

What happened?
I am running a cluster using kops version 1.15.2 on AWS. The default storageclass is gp2. Last week we enabled the EBS CSI driver per the docs in this repository. We did not enable the CSIMigration or CSIMigrationAWS feature gates as we are not yet ready to migrate over the "in-tree" storageclass (gp2). While the driver itself works when we create a PVC referencing a StorageClass similar to this where the provisioner is set to ebs.csi.aws.com we also are seeing an unintended side-effect of duplicate persistent volumes being created whenever we create a PVC referencing the "in tree" (gp2) storage class. Specifically, the "in tree" storage class creates on PV and the AWS EBS CSI driver creates another. I can see in the logs that the AWS EBS CSI driver is creating the PV, even though we have not migrated the "in-tree" StorageClass to use the CSI driver:

[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.067924       1 controller.go:979] Final error received, removing PVC a6767ac2-340e-42ad-8d59-35cc744d0da6 from claims in progress 
[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.067944       1 controller.go:902] Provisioning succeeded, removing PVC a6767ac2-340e-42ad-8d59-35cc744d0da6 from claims in progress 
[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.072531       1 controller.go:1196] provision "default/orth-pvc-test" class "gp2": started 
[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.075212       1 controller.go:367] translating storage class for in-tree plugin kubernetes.io/aws-ebs to CSI 
[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.075309       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"orth-pvc-test", UID:"a6767ac2-340e-42ad-8d59-35cc744d0da6", APIVersion:"v1", ResourceVersion:"102811599", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/orth-pvc-test" 
[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.086061       1 controller.go:471] CreateVolumeRequest {Name:pvc-a6767ac2-340e-42ad-8d59-35cc744d0da6 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[type:gp2] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:requisite:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-west-2a" > > requisite:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-west-2b" > > preferred:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-west-2b" > > preferred:<segments:<key:"topology.ebs.csi.aws.com/zone" value:"us-west-2a" > >  XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} 
[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.086139       1 connection.go:180] GRPC call: /csi.v1.Controller/CreateVolume 
[ebs-csi-controller-0 csi-provisioner] I0225 16:55:22.086144       1 connection.go:181] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-west-2b"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-west-2a"}}],"requisite":[{"segments":{"topology.ebs.csi.aws.com/zone":"us-west-2a"}},{"segments":{"topology.ebs.csi.aws.com/zone":"us-west-2b"}}]},"capacity_range":{"required_bytes":1073741824},"name":"pvc-a6767ac2-340e-42ad-8d59-35cc744d0da6","parameters":{"type":"gp2"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]} 

What you expected to happen?
Not create duplicate volumes when using in-tree storage class (gp2).

How to reproduce it (as minimally and precisely as possible)?
Deploy a cluster via kops to AWS using version 1.15.2. Deploy the AWS EBS CSI driver to this cluster following the instructions on the README.md in this repository.

Create a new PVC similar to this:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: orth-pvc-test
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp2

Log into the AWS console and view the EBS volumes. Notice there are two volumes created at almost the exact same time - both have tags referencing the Kubernetes PersistentVolume.

Anything else we need to know?:
When this situation occurs, it seems that the majority of the time, the EBS volume provisioned by the "in-tree" storage class is the one that actually gets attached to the Kubernetes persistent volume. However, there have been situations where the volume provisioned by the EBS CSI driver is the one that is attached.

Also, here is what the gp2 storage class looks like on this cluster, for reference:

$ kubectl get storageclass gp2 -o yaml

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"storageclass.beta.kubernetes.io/is-default-class":"true"},"creationTimestamp":"2019-02-21T23:54:20Z","labels":{"k8s-addon":"storage-aws.addons.k8s.io"},"name":"gp2","resourceVersion":"128983847","selfLink":"/apis/storage.k8s.io/v1/storageclasses/gp2","uid":"018de370-3634-11e9-8036-02cea187b3be"},"parameters":{"type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2019-02-21T23:54:20Z"
  labels:
    k8s-addon: storage-aws.addons.k8s.io
  name: gp2
  resourceVersion: "131391863"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
  uid: 018de370-3634-11e9-8036-02cea187b3be
parameters:
  type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Delete
volumeBindingMode: Immediate

Environment

• Kubernetes version (use kubectl version):
  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

• Driver version:
  amazon/aws-ebs-csi-driver:v0.4.0

[leakingtapan]

This issue is fixed by this: kubernetes-csi/external-provisioner@3184db6

And we are picking up 1.5.0 external provisioner in the next release

[ryaneorth]

Perfect, thank you @leakingtapan for the prompt response! When do you plan to do the next release?

And is there any work-around in the meantime?

[ryaneorth]

It looks like the bug-fix for this was back-ported to the external-provisioner version 1.3.1 so I assume I can update to use that in the meantime:

kubernetes-csi/external-provisioner@263a709

[aviau]

I am getting this issue with with K8S 1.15.11 on EKS, how does one change the external provisioner version to one with the fix?

Edit: my mistake! I had the provisioner field set to kubernetes.io/aws-ebs in the storageclass. However maybe this is still a bug? Anyways, I switched to ebs.csi.aws.com and it works.

[pandvan]

Kustomization file on your master branch is still referencing to version 1.3.0 of csi-provisioner (https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/deploy/kubernetes/overlays/stable/kustomization.yaml).
I'm still facing the issue using versione 0.5.0 of ebs-csi-driver.
How can I update my deployment in order to use a fixed version of the provisioner?