doc: adminid and userid not documented #80

pohly · 2018-10-02T11:28:08Z

examples/rbd/storageclass.yaml contains these settings:

    # Ceph users for operating RBD
    adminid: admin
    userid: kubernetes

Those are not mentioned under https://github.com/ceph/ceph-csi/blob/master/docs/deploy-rbd.md#configuration. Are both users needed?

I found the entire "Required secrets" section a bit confusing:

the value is its password - is that really called password in Ceph? I think this refers to the base64 encoded key that is stored in ceph keyrings, right? Linking to http://docs.ceph.com/docs/mimic/rbd/rados-rbd-cmds/#create-a-block-device-user might be useful here, with a few words how to use the result.
CSI RBD expects admin keyring and Ceph config file in /etc/ceph - how can that be achieved when deplying in Kubernetes? Is it really necessary?

I have set up a test cluster, but haven't actually tried provisioning. I guess I'll find out soon whether I interpreted the instructions correctly ;-}

The text was updated successfully, but these errors were encountered:

pohly · 2018-10-02T11:39:27Z

Nope, looks like I didn't populate my secret correctly. The external-provisioner is sent binary data for the secrets and then complains (these are disposable secrets, so I can log them as they are):

I1002 11:34:43.585220       1 controller.go:104] GRPC request: name:"pvc-f7aaf95ec63611e8" capacity_range:<required_bytes:1073741824 > volume_capabilities:<mount:<> access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:"adminid" value:"admin" > parameters:<key:"csiNodePublishSecretName" value:"csi-rbd-secret" > parameters:<key:"csiNodePublishSecretNamespace" value:"default" > parameters:<key:"csiProvisionerSecretName" value:"csi-rbd-secret" > parameters:<key:"csiProvisionerSecretNamespace" value:"default" > parameters:<key:"imageFeatures" value:"layering" > parameters:<key:"imageFormat" value:"2" > parameters:<key:"monitors" value:"192.168.7.2:6789,192.168.7.4:6789,192.168.7.6:6789,192.168.7.8:6789" > parameters:<key:"pool" value:"rbd" > parameters:<key:"userid" value:"kubernetes" > controller_create_secrets:<key:"admin" value:"\001\000{'\263[e\353\230\010\020\000\026\366\022U\233\027f\014\027\201=\211:}\204\331" > controller_create_secrets:<key:"kubernetes" value:"\001\000\227R\263[\246^\027\n\020\000\177\016\374\326\241AK\005\332\201\252\333b\361\\\n" > 
I1002 11:34:43.585422       1 controller.go:106] GRPC response: 
I1002 11:34:43.585447       1 controller.go:107] GRPC error: rpc error: code = Internal desc = grpc: error while marshaling: proto: invalid UTF-8 string
E1002 11:34:43.585482       1 controller.go:895] Failed to provision volume for claim "default/rbd-pvc" with StorageClass "csi-rbd": rpc error: code = Internal desc = grpc: error while marshaling: proto: invalid UTF-8 string

pohly · 2018-10-02T11:48:59Z

I got provisioning working by storing the string from the keyring (for example, AQCXUrNbpl4XChAAfw781qFBSwXagarbYvFcCg==) encoded once more with base64 in the secret (QVFDWFVyTmJwbDRYQ2hBQWZ3NzgxcUZCU3dYYWdhcmJZdkZjQ2c9PQo=).

pohly · 2018-10-02T14:59:13Z

Another pitfall was that --kubelet-registration-path mode for the driver-registrar in https://github.com/ceph/ceph-csi/blob/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml only works when the kubelet supports registration, which apparently wasn't the case on my 1.11 cluster. https://github.com/kubernetes-csi/docs/pull/55/files is open and will have more information about this requirement.

I got that resolved by removing the --kubelet-registration-path. Now I did get PublishVolume called in the right host, but it fails:

I1002 14:48:08.088931       1 utils.go:96] GRPC call: /csi.v0.Node/NodePublishVolume
I1002 14:48:08.088951       1 utils.go:97] GRPC request: volume_id:"csi-rbd-b2c8f7c7-c64e-11e8-9890-deadbeef0101" target_path:"/var/lib/kubelet/pods/7591f70e-c64f-11e8-a8d5-deadbeef0100/volumes/kubernetes.io~csi/pvc-afb91735c64e11e8/mount" volume_capability:<mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > > node_publish_secrets:<key:"admin" value:"AQB7J7NbZeuYCBAAFvYSVZsXZgwXgT2JOn2E2Q==\n" > node_publish_secrets:<key:"kubernetes" value:"AQCXUrNbpl4XChAAfw781qFBSwXagarbYvFcCg==\n" > volume_attributes:<key:"adminid" value:"admin" > volume_attributes:<key:"csiNodePublishSecretName" value:"csi-rbd-secret" > volume_attributes:<key:"csiNodePublishSecretNamespace" value:"default" > volume_attributes:<key:"csiProvisionerSecretName" value:"csi-rbd-secret" > volume_attributes:<key:"csiProvisionerSecretNamespace" value:"default" > volume_attributes:<key:"imageFeatures" value:"layering" > volume_attributes:<key:"imageFormat" value:"2" > volume_attributes:<key:"monitors" value:"192.168.7.2:6789,192.168.7.4:6789,192.168.7.6:6789,192.168.7.8:6789" > volume_attributes:<key:"pool" value:"rbd" > volume_attributes:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1538478360538-8081-csi-rbdplugin" > volume_attributes:<key:"userid" value:"kubernetes" > 
I1002 14:48:08.090717       1 mount_linux.go:209] Cannot run systemd-run, assuming non-systemd OS
I1002 14:48:08.090737       1 mount_linux.go:210] systemd-run failed with: exit status 1
I1002 14:48:08.090746       1 mount_linux.go:211] systemd-run output: Failed to create bus connection: No such file or directory
I1002 14:48:08.090801       1 keymutex.go:50] LockKey(...) called for id "rbd/pvc-afb91735c64e11e8"
I1002 14:48:08.090816       1 keymutex.go:53] LockKey(...) for id "rbd/pvc-afb91735c64e11e8" completed.
I1002 14:48:08.092263       1 rbd_util.go:159] rbd: status pvc-afb91735c64e11e8 using mon 192.168.7.2:6789,192.168.7.4:6789,192.168.7.6:6789,192.168.7.8:6789, pool rbd id kubernetes key AQCXUrNbpl4XChAAfw781qFBSwXagarbYvFcCg==
I1002 14:48:18.172392       1 keymutex.go:59] UnlockKey(...) called for id "rbd/pvc-afb91735c64e11e8"
I1002 14:48:18.172451       1 keymutex.go:66] UnlockKey(...) for id. Mutex found, trying to unlock it. "rbd/pvc-afb91735c64e11e8"
I1002 14:48:18.172462       1 keymutex.go:69] UnlockKey(...) for id "rbd/pvc-afb91735c64e11e8" completed.
E1002 14:48:18.172474       1 utils.go:100] GRPC error: fail to check rbd image status with: (exit status 1), rbd output: (2018-10-02 14:48:08.110153 7f0286e44d40 -1 did not load config file, using default settings.
2018-10-02 14:48:08.115699 7f0286e44d40 -1 Errors while parsing config file!
2018-10-02 14:48:08.115709 7f0286e44d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-10-02 14:48:08.115710 7f0286e44d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-10-02 14:48:08.115711 7f0286e44d40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2018-10-02 14:48:08.117466 7f0286e44d40 -1 Errors while parsing config file!
2018-10-02 14:48:08.117477 7f0286e44d40 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2018-10-02 14:48:08.117477 7f0286e44d40 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2018-10-02 14:48:08.117478 7f0286e44d40 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
2018-10-02 14:48:08.149791 7f0286e44d40 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.kubernetes.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
2018-10-02 14:48:18.165976 7f0261ffb700 -1 librbd::image::OpenRequest: failed to stat v2 image header: (1) Operation not permitted
2018-10-02 14:48:18.166125 7f02617fa700 -1 librbd::ImageState: 0x55a21c5f9b80 failed to open image: (1) Operation not permitted
rbd: error opening image pvc-afb91735c64e11e8: (1) Operation not permitted

Is that the point where I somehow have to create /etc/ceph with a suitable keyring? But where (host or container) and how (if its inside the container, where the CSI driver runs)?

rootfs · 2018-10-02T15:07:05Z

that /etc/ceph.conf message is benign, rbd gets all config from cli options, not from ceph.conf

can you check if id kubernetes has access to rbd pool? ceph auth ls output will help

rootfs · 2018-10-02T15:10:06Z

btw, k8s 1.11 has feature gate against driver probe, while 1.12 turns on the feature by default.

pohly · 2018-10-02T15:27:32Z

ceph auth ls shows that there is an entry for kubernetes:

...
client.kubernetes
	key: AQCXUrNbpl4XChAAfw781qFBSwXagarbYvFcCg==
	caps: [mon] profile rbd
	caps: [osd] profile rbd pool=rdb

According to my understanding of http://docs.ceph.com/docs/mimic/rbd/rados-rbd-cmds/#create-a-block-device-user, that should grant read/write access to the pool, right?

So admin is used for provisioning and kubernetes for publishing?

dillaman · 2018-10-02T15:32:21Z

@pohly Is that a typo on your pool=rdb?

pohly · 2018-10-02T15:37:05Z

@dillaman that's copied straight from the ceph auth ls output, so no, not a typo. The example from http://docs.ceph.com/docs/mimic/rbd/rados-rbd-cmds/#create-a-block-device-user also uses profile rbd pool=vms. Is that wrong?

pohly · 2018-10-02T15:39:19Z

Here's the command I used for creating the user:

ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=rdb'

dillaman · 2018-10-02T15:40:15Z

@pohly Your RBD pool is really named "rdb" and not "rbd"?

pohly · 2018-10-02T15:40:50Z

@dillaman Ooops! You are right of course. Thanks for spotting this.

pohly · 2018-10-05T18:25:50Z

I don't want to abuse this issue here for support questions, but I am again seeing something related to keys. I promise to make up for all the answers here by creating a doc PR ;-}

I have configured the E2E test in Kubernetes to run against ceph-csi on a local cluster. It gets to the point where an rbd volume was provisioned, but then formatting the volume seems to fail after attaching it to /dev/rbd0. Attempts to delete the volume fail with:

csi-rbdplugin-xm2wl/csi-rbdplugin: I1005 18:16:08.748905       1 utils.go:96] GRPC call: /csi.v0.Controller/DeleteVolume
csi-rbdplugin-xm2wl/csi-rbdplugin: I1005 18:16:08.748917       1 utils.go:97] GRPC request: volume_id:"csi-rbd-ac3a2d38-c8c9-11e8-92fb-deadbeef0101" 
csi-rbdplugin-provisioner-0/csi-provisioner: I1005 18:16:08.749163       1 controller.go:107] GRPC error: rpc error: code = Unknown desc = RBD key for ID: admin not found
csi-rbdplugin-provisioner-0/csi-provisioner: E1005 18:16:08.749174       1 controller.go:1138] Deletion of volume "pvc-631d4a71c8c911e8" failed: rpc error: code = Unknown desc = RBD key for ID: admin not found
csi-rbdplugin-provisioner-0/csi-provisioner: E1005 18:16:08.749218       1 goroutinemap.go:150] Operation for "delete-pvc-631d4a71c8c911e8[b52f9a01-c8c9-11e8-b45b-deadbeef0100]" failed. No retries permitted until 2018-10-05 18:17:12.749201098 +0000 UTC m=+230.079890436 (durationBeforeRetry 1m4s). Error: "rpc error: code = Unknown desc = RBD key for ID: admin not found"
csi-rbdplugin-xm2wl/csi-rbdplugin: I1005 18:16:08.748984       1 controllerserver.go:163] deleting volume pvc-631d4a71c8c911e8
csi-rbdplugin-xm2wl/csi-rbdplugin: I1005 18:16:08.748992       1 controllerserver.go:165] failed to delete rbd image: rbd/pvc-631d4a71c8c911e8 with error: RBD key for ID: admin not found
csi-rbdplugin-xm2wl/csi-rbdplugin: E1005 18:16:08.748998       1 utils.go:100] GRPC error: RBD key for ID: admin not found

This I find a bit strange: the controller server must have been able to create the volume, why does deleting now fail?

rootfs · 2018-10-05T18:36:43Z

@mkimuram has a k8s e2e PR using rbd csi driver

mkimuram · 2018-10-05T19:56:16Z

Please see kubernetes/kubernetes#67088

Electronn · 2018-11-14T12:11:56Z

examples/rbd/storageclass.yaml contains these settings:
    # Ceph users for operating RBD
    adminid: admin
    userid: kubernetes
Those are not mentioned under https://github.com/ceph/ceph-csi/blob/master/docs/deploy-rbd.md#configuration. Are both users needed?

I found the entire "Required secrets" section a bit confusing:

the value is its password - is that really called password in Ceph? I think this refers to the base64 encoded key that is stored in ceph keyrings, right? Linking to http://docs.ceph.com/docs/mimic/rbd/rados-rbd-cmds/#create-a-block-device-user might be useful here, with a few words how to use the result.

CSI RBD expects admin keyring and Ceph config file in /etc/ceph - how can that be achieved when deplying in Kubernetes? Is it really necessary?

I have set up a test cluster, but haven't actually tried provisioning. I guess I'll find out soon whether I interpreted the instructions correctly ;-}

WOW . You save a lot of my time. We need to base64 encode one more time our ceph client user key.

kfox1111 · 2018-11-15T00:00:11Z

We got this working, but have a question. I don't see anywhere in the code, where it actually needs a full blown cluster admin credential as opposed to just a credential that has access just to the pool the provisioner will be creating images in. We'd much rather give it the more restricted account if possible as we have multiple clusters targeting the same ceph. Is this safe? This is a config similar to other systems like openstack cinder use.

rootfs · 2018-11-15T13:16:30Z

@kfox1111 that makes sense. the admin credentials only needs to have caps to create images. If you have multiple clusters and want to restrict admin accounts, you have a per pool admin (refer to this doc and use that pool admin instead of ceph admin to provision images.

kfox1111 · 2018-11-15T18:17:59Z

so you would use the same account for user and admin, or would the user account be further restricted?

rootfs · 2018-11-15T19:25:37Z

rbd only allows one user and admin per storage class. The user id is not dynamically generated (as in cephfs).

One way (not meant to be best practice) is to partition your ceph storage into multiple pools, one for each of your kubernetes cluster or namespaces. You can restrict admin id to have privileged osd caps per pool, while limit such caps for user id in that pool.

pohly · 2018-11-15T19:59:47Z

Huamin Chen <notifications@github.com> writes:

rbd only allows one user and admin per storage class. The user id is not dynamically generated (as in cephfs).

Note that the storage class can use templates for the secret name and namespace. Then the actual secrets can be supplied by the user who creates the PVC: https://kubernetes-csi.github.io/docs/Usage.html#csi-provisioner-parameters

kfox1111 · 2018-11-16T00:53:44Z

@rootfs ok, thanks.

@pohly thats very interesting. Thanks for the link. :)

gman0 · 2018-11-29T21:56:46Z

Regarding the need of encoding the keys in base64: that's more of just a quirk in Kubernetes where users creating Secret objects manually with kubectl create need to pass the values encoded in base64 (for example, they wouldn't have to do this extra step with --from-literal). It's all documented in the official Kubernetes docs for Secrets.

Madhu-1 · 2019-07-16T14:27:46Z

closing this one as we moved using base64 encoded userID and password to use plain text in secret.
#395

sync downstream devel with upstream devel

pohly mentioned this issue Oct 2, 2018

Doc Update for Registrar Sidecar Setup with Kubelet Plugin Registration kubernetes-csi/docs#55

Merged

pohly mentioned this issue Feb 21, 2019

Allow PVC name/namespace as template for provisioner-secret kubernetes-csi/external-provisioner#233

Closed

ShyamsundarR mentioned this issue Mar 20, 2019

Suggested changes to ID and key requirements for rbd CSI plugin #270

Closed

ShyamsundarR mentioned this issue Jun 22, 2019

Modify RBD plugin to use a single ID and move the id and key into the… #395

Merged

Madhu-1 closed this as completed Jul 16, 2019

Rakshith-R referenced this issue in Rakshith-R/ceph-csi May 26, 2022

Merge pull request #80 from ceph/devel

000bb32

sync downstream devel with upstream devel

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: adminid and userid not documented #80

doc: adminid and userid not documented #80

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

rootfs commented Oct 2, 2018

rootfs commented Oct 2, 2018

pohly commented Oct 2, 2018

dillaman commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

dillaman commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 5, 2018

rootfs commented Oct 5, 2018

mkimuram commented Oct 5, 2018

Electronn commented Nov 14, 2018

kfox1111 commented Nov 15, 2018

rootfs commented Nov 15, 2018

kfox1111 commented Nov 15, 2018

rootfs commented Nov 15, 2018

pohly commented Nov 15, 2018 via email

kfox1111 commented Nov 16, 2018

gman0 commented Nov 29, 2018

Madhu-1 commented Jul 16, 2019

doc: adminid and userid not documented #80

doc: adminid and userid not documented #80

Comments

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

rootfs commented Oct 2, 2018

rootfs commented Oct 2, 2018

pohly commented Oct 2, 2018

dillaman commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 2, 2018

dillaman commented Oct 2, 2018

pohly commented Oct 2, 2018

pohly commented Oct 5, 2018

rootfs commented Oct 5, 2018

mkimuram commented Oct 5, 2018

Electronn commented Nov 14, 2018

kfox1111 commented Nov 15, 2018

rootfs commented Nov 15, 2018

kfox1111 commented Nov 15, 2018

rootfs commented Nov 15, 2018

pohly commented Nov 15, 2018 via email

kfox1111 commented Nov 16, 2018

gman0 commented Nov 29, 2018

Madhu-1 commented Jul 16, 2019