Raw block volume support #805

erinboyd · 2017-07-12T18:57:02Z

Design proposal for raw block via PV

feiskyer · 2017-07-13T07:44:55Z

@erinboyd Seems the proposal is overlapping with local persistent storage, which including features like local block devices as a volume and local raw block device.

fabiand

This proposal also looks pretty rough, and I'm missing details about how the raw block storage is now envisioned.

fabiand · 2017-07-17T10:53:13Z

contributors/design-proposals/raw-block-pv.md

+
+Authors: erinboyd@, screeley44@, mtanino@
+
+This document presents a proposal for managing raw block storage in Kubernetes using the persistent volume soruce API as a consistent model


fabiand · 2017-07-17T10:54:42Z

contributors/design-proposals/raw-block-pv.md

+  Before the advent of storage plugins, emptyDir and hostPath were widely used to quickly prototype stateless applications in Kube. 
+  Both have limitations for their use in application that need to store persistent data or state. 
+  EmptyDir, though quick and easy to use, provides no real guarantee of persistence for suitable amount of time. 
+  Appropriately used as scratch space, it does not have the HostPath, became an initial offering for local storage, but had many drawbacks. Without having the ability to guarantee space & ensure ownership, one would lose data once a node was rescheduled. Therefore, the risk outweighed the reward when trying to  leverage the power of local storage needed for stateful applications like databases.


erinboyd · 2017-07-21T23:00:38Z

@feiskyer @fabiand The local block devices mentioned raw devices are out of scope. The raw block device changes being proposed are for both local and network attached raw devices.

erinboyd · 2017-07-24T12:09:29Z

@screeley44 @msau42 @mtanino please review

ianchakeres · 2017-07-25T17:19:41Z

contributors/design-proposals/raw-block-pv.md

+  drawbacks. Without having the ability to guarantee space & ensure ownership, one would lose data once a node was rescheduled. 
+  Therefore, the risk outweighed the reward when trying to  leverage the power of local storage needed for stateful applications like 
+  databases.
+


I'm not sure that many people are familiar with the difference between a raw block volume and a block volume with a filesystem on it.

Raw block will provide the ability to perform block operations on the volumes, which are required or desired by several stateful systems. For example BlueStore/BlueFS for Ceph and MySQL

@ianchakeres
Agree. We need to define terminologies on top of the document.

There's a 'system utility' & bootstrap use case that should be noted. If a fs health check, partition/format or other disk utility needs to run on the node it will want raw block. Examples may be local storage partition & format of a raw disk, deploy kube to a bare metal node, utility pods that check disk health after reboot etc..

mtanino · 2017-07-25T19:33:35Z

contributors/design-proposals/raw-block-pv.md

@@ -0,0 +1,571 @@
+# Local Raw Block Consumption via Persistent Volume Source


We can remove "local" from the title since both local and network attached are target of this proposal.
Can we use same terminology throughout the spec?

block volume

raw block volume

raw block device

I think we don't need "via Persistent Volume Source" on the title since this spec will cover both static and dynamic provisioning, right?

I think it's implying that we're not going to support direct volume access (ie VolumeSource in the pod)

I got an intention. thanks.

So, I noticed someone changed the title to just raw block, are we fine with this?

I am fine with it. The changes proposed here can be applied to any volume plugin, not just local storage.

@erinboyd
I mean the title of committed document would be "Raw Block Volume Consumption via Persistent Volume Source" or something like that. At least, please remove the "local".
I think the title of this PR is originally "Raw block". I'd prefer the title of this PR like "[Proposal] Raw block volume support" but this is nit.

mtanino · 2017-07-25T19:35:27Z

contributors/design-proposals/raw-block-pv.md

+
+# Value add to Kubernetes
+
+  Before the advent of storage plugins, emptyDir and hostPath were widely used to quickly prototype stateless applications in Kube. 


s/Kube/Kubernetes/

mtanino · 2017-07-25T19:39:05Z

contributors/design-proposals/raw-block-pv.md

+* Enable durable access to block storage
+* Support storage requirements for all workloads supported by Kubernetes
+* Provide flexibility for users/vendors to utilize various types of storage devices
+* Agree on API changes for block


block volume support
or raw block volume support
or raw block device support

This goal is already implied by "Enable durable access to block storage"

mtanino · 2017-07-25T19:55:05Z

contributors/design-proposals/raw-block-pv.md

+
+DESCRIPTION: 
+
+A developer wishes to enable their application to use a local raw block device as the volume for the container. The admin has already created PVs that the user will bind to by specifying 'block' as the volume type of their PVC.


I think iSCSI or FC are type of network attached.
I suppose that the local storage means the server itself has own physical storage like SSDs inside the server.

I recommend to add terminology section to define terms such as local storage, network attached storage at the top of this document to avoid confusion.

Yes, this example is a little confusing because it mentions local storage and iscsi/fc together.

Currently, most of storage plugins are network attached, I think we should introduce examples mainly base on network attached storage plugin such as gce, aws, fc, iscsi, nfs etc, and then we can add local storage plugin(local storage case) as additional examples, but we need clearly distinguish between network attached case and local storage case to avoid confusion.

mtanino · 2017-07-25T19:58:49Z

@saad-ali

mtanino · 2017-07-25T20:02:10Z

contributors/design-proposals/raw-block-pv.md

+ 	Unmounter
+ 	GetVolumePath() string
+}
+```


Please add "Features & Milestones" section, then we need to mention that there are two phases to support of block volume support.

mtanino · 2017-07-25T20:03:56Z

contributors/design-proposals/raw-block-pv.md

@@ -0,0 +1,571 @@
+# Local Raw Block Consumption via Persistent Volume Source


I think we don't need "via Persistent Volume Source" on the title since this spec will cover both static and dynamic provisioning, right?

screeley44 · 2017-07-25T20:39:40Z

contributors/design-proposals/raw-block-pv.md

+  accessModes:
+    - "ReadWriteOnce"
+  gcePersistentDisk:
+    fsType: "raw"


fsType: "raw" will also require a change in mount_linux.go for formatAndMount method correct? If so probably want to call this out as a change in the proposal like you do for the api changes in the yaml

Updated in the milestones and features

Instead of special casing "raw" as a value for fsType, can fsType just be ignored/empty if volumeType == block?

Could there be a requested volume (other than GCE) where you would request volumeType: block and also want a fs put on...seems if we always ignore fsType when block is requested we could have a problem for other storage plugins or even future use cases?

+1 to @msau42, if someone wants a raw device then volume plugin should ignore fsType part of the PV. If some crazy pod wants a raw device with a filesystem on it the pod can call mkfs on its own.

Mounter binding matrix for "Dynamic Provisioning"

# Plugin fstype PVC volumeType Result Result of volume

00 unspecified BIND file(ext4)

01 block BIND block

@mtanino @jsafrane @msau42 so there is much discussion around not having an fstype. So what if you aren't pre-creating the volumes (hence the PV doesn't exist). You would only have the PVC and the storageClass to indicate block. You could potentially have a provisioner that then creates the block device and adds a filesystem or not based on the fstype. Thoughts?

Like @jsafrane mentioned, today provisioners don't format the filesystem. It's only at mount time that the plugin formats a filesystem. So I don't think provisioners or binding should use fstype, but they should use volumeType

@jsafrane @msau42 ^^^^ don't you think it makes sense to include it given the above? Using the RO boolean wouldn't work either as RBD aren't RO.
Please let us know.

Sorry I'm not sure which above you're referring to. Can you clarify?

msau42 · 2017-07-25T20:28:43Z

contributors/design-proposals/raw-block-pv.md

@@ -0,0 +1,571 @@
+# Local Raw Block Consumption via Persistent Volume Source


I think it's implying that we're not going to support direct volume access (ie VolumeSource in the pod)

msau42 · 2017-07-25T20:29:30Z

contributors/design-proposals/raw-block-pv.md

+* Provide flexibility for users/vendors to utilize various types of storage devices
+* Agree on API changes for block
+* Provide a consistent security model for block devices 
+* Provide block storage usage isolation


What do you mean by isolation?

Basically to only allow one user to use the raw block device as done with fsgroup today

Can you clarify that here? There are other kinds of isolation that we sometimes talk about, like capacity and IOPS isolation.

I don't think we are guaranteeing IOPS or capacity isolation. I will be explicit on this bullet point

msau42 · 2017-07-25T20:32:34Z

contributors/design-proposals/raw-block-pv.md

+* Support all storage devices natively in upstream Kubernetes. Non-standard storage devices are expected to be managed using extension
+  mechanisms.
+
+# Value add to Kubernetes


This section talks about ephemeral vs persistent, and downsides of hostpath volumes for local storage. But it doesn't really answer why block is valuable.

This has been updated, please review

msau42 · 2017-07-25T20:34:41Z

contributors/design-proposals/raw-block-pv.md

+  The additional benefit of explicitly defining how the volume is to be consumed will provide a means for indicating the method
+  by which the device should be scrubbed when the claim is deleted, as this method will differ from a raw block device compared to a 
+  filesystem. The ownership of scrubbing the device properly shall be up to the plugin method being utilized.
+  The last design point is block devices should be able to be fully restricted by the admin in accordance with how inline volumes 


I opened up issue kubernetes/kubernetes#44982 about this. Currently pod security policy only enforces direct volume sources, and not Persistent Volumes.

msau42 · 2017-07-25T20:35:31Z

contributors/design-proposals/raw-block-pv.md

+
+## Persistent Volume Claim API Changes:
+In the simplest case of static provisioning, a user asks for a volumeType of block. The binder will only bind to a PV defined 
+with the same label.


s/label/volumeType

msau42 · 2017-07-25T20:42:29Z

contributors/design-proposals/raw-block-pv.md

+
+DESCRIPTION: 
+
+A developer wishes to enable their application to use a local raw block device as the volume for the container. The admin has already created PVs that the user will bind to by specifying 'block' as the volume type of their PVC.


Yes, this example is a little confusing because it mentions local storage and iscsi/fc together.

msau42 · 2017-07-25T20:44:12Z

contributors/design-proposals/raw-block-pv.md

+	claimName: raw-pvc
+```
+
+## UC3: 


How is this different from UC1?

Updated to have UC1 admin focused and UC3 to be user focused

msau42 · 2017-07-25T20:46:14Z

contributors/design-proposals/raw-block-pv.md

+metadata:
+  name: web
+spec:
+  serviceName: "nginx"


I know the actual gluster spec may be more complicated than this, but it may fit the example better if we used gluster here instead of nginx.

And then have an example of an application pod using gluster volumes?

This is UC7

Yeah, so the example in UC7 shows a nginx application using ebs raw block volumes.

I think what you actually want, is an gluster application using ebs raw block volumes, and then an nginx application using gluster volumes.

@msau42 this example was provided by CoreOS (@lpabon) which utilizes QuarterMaster to then deploy gluster on top of the dynamically provisioned EBS volumes. We can either include this process as part of the workflow or I am up for suggestions how to better clarify how things work today

@lpabon can you clarify the example here? My understanding is that the volume access would be like: application -> gluster -> raw block

@lpabon ^^^

@msau42 I think what you actually want, is an gluster application using ebs raw block volumes, and then an nginx application using gluster volumes. -- Correct.

The goal here is for raw block devices to be created using dynamic provisioning, then attached to the host/container for access -- without a file system being created on top of it.

I think we can remove the nginx example. It could create confusion.

@lpabon so just remove UC7 all together?

@erinboyd Please don't :) . Just remove the nginx example. I will provide an example for you

msau42 · 2017-07-25T20:47:12Z

contributors/design-proposals/raw-block-pv.md

+
+## UC9: 
+
+DESCRIPTION: Developer wishes to consumes raw device


How is this different from UC1?

Removed as dupicate @msau42

msau42 · 2017-07-25T20:47:38Z

contributors/design-proposals/raw-block-pv.md

+
+# Implementation Plan
+
+Phase 1: Pre-provisioned PVs to precreated devices


Also CRI changes

mtanino · 2017-07-26T15:38:24Z

contributors/design-proposals/raw-block-pv.md

+provisioner: no-provisioning 
+parameters:
+  volumeType: block
+


please remove empty line.

mtanino · 2017-07-26T15:38:31Z

contributors/design-proposals/raw-block-pv.md

+   - ReadWriteOnce
+ local:
+    path: /dev/xvdc
+


please remove empty line.

mtanino · 2017-07-26T15:38:36Z

contributors/design-proposals/raw-block-pv.md

+provisioner: kubernetes.io/local-block-ssd
+parameters:
+  volumeType: block
+


please remove empty line.

mtanino · 2017-07-26T15:38:43Z

contributors/design-proposals/raw-block-pv.md

+ resources:
+   requests:
+     storage: 10Gi
+


please remove empty line.

mtanino · 2017-07-26T15:38:56Z

contributors/design-proposals/raw-block-pv.md

+  gcePersistentDisk:
+    fsType: "raw"
+    pdName: "gce-disk-1"
+


please remove empty line.

mtanino · 2017-07-26T15:48:47Z

contributors/design-proposals/raw-block-pv.md

+
+DESCRIPTION: 
+
+A developer wishes to enable their application to use a local raw block device as the volume for the container. The admin has already created PVs that the user will bind to by specifying 'block' as the volume type of their PVC.


Currently, most of storage plugins are network attached, I think we should introduce examples mainly base on network attached storage plugin such as gce, aws, fc, iscsi, nfs etc, and then we can add local storage plugin(local storage case) as additional examples, but we need clearly distinguish between network attached case and local storage case to avoid confusion.

mtanino · 2017-07-26T16:29:55Z

To access this proposal easily, could you update the title from "Raw block"
to like "[Proposal] Raw block volume support"?

gnufied · 2017-07-31T18:07:09Z

contributors/design-proposals/raw-block-pv.md

+metadata:
+name: myclaim
+spec:
+**volumeType: block**


This doesn't render correctly if the entire block is formatted. how about we simply add a comment "# new field"

erinboyd · 2017-08-03T14:51:42Z

@k8s-sig-api @k8s-sig-apps @k8s-sig-node

erinboyd · 2017-08-03T14:52:11Z

@msau42 @mtanino @screeley44 please re-review changes according to comments

mtanino · 2017-08-03T15:11:57Z

contributors/design-proposals/raw-block-pv.md

@@ -0,0 +1,559 @@
+# Local Raw Block Consumption in Kubernetes


Please remove "Local".
~~Local~~ Raw Block Consumption in Kubernetes

@mtanino done

mtanino · 2017-08-03T15:16:02Z

contributors/design-proposals/raw-block-pv.md

+provisioner: no-provisioning 
+parameters:
+```
+


Could you add "volumeType combination matrix" which we have on the google doc?
User and Admin need to know the combination of parameters between pv, pvc and storageclass.

@mtanino Added

jsafrane · 2017-08-03T17:00:53Z

/cc

jsafrane · 2017-08-03T17:02:15Z

/sig storage

msau42 · 2017-08-03T20:39:49Z

contributors/design-proposals/raw-block-pv.md

+  Therefore, the risk outweighed the reward when trying to  leverage the power of local storage needed for stateful applications like 
+  databases.
+
+  By extending the API for volumes to specifically request a raw block device, we provide a consist method for accessing volumes. In


How was it not consistent before?

By having the raw block devices created as PVs they are more consumable rather than in terms of how file on block is handled today, the raw devices are created outside of kube and kube has no knowledge or reference to what is available.

I guess my confusion is that from a user point of view, the access of volume's today is consistent. It's always a filesystem.

Updated to make it more concise

msau42 · 2017-08-03T20:40:32Z

contributors/design-proposals/raw-block-pv.md

+
+  By extending the API for volumes to specifically request a raw block device, we provide a consist method for accessing volumes. In
+  addition, the ability to use a raw block device without a filesystem will allow Kuberenets better support of high performance 
+  applications that can utilitze raw block devices directly for their storage.


Can you give some examples here?

I think these are covered in the Use Cases. I don't see a need to duplicate it here unless you feel it's necessary

I don't mean yaml examples, I mean just naming some of the DBs that could benefit from it.

@msau42 added some examples, but as said above, these are covered extensively in the use cases

msau42 · 2017-08-03T20:41:14Z

contributors/design-proposals/raw-block-pv.md

+
+# Value add to Kubernetes
+
+  Before the advent of storage plugins, emptyDir and hostPath were widely used to quickly prototype stateless applications in Kubernetes. 


Not sure if we need the all this storage history.

I would like it in there...code is 90% history

msau42 · 2017-08-03T20:43:40Z

contributors/design-proposals/raw-block-pv.md

+
+## Persistent Volume API Changes:
+For static provisioning the admin creates the volume and also is intentional about how the volume should be consumed. For backwards
+compatibility, the absence of volumeType will default to volumes work today, which are formatted with a filesystem depending on 


default to file, which is how volumes work today

@msau42 fixed

msau42 · 2017-08-03T20:44:53Z

contributors/design-proposals/raw-block-pv.md

+For static provisioning the admin creates the volume and also is intentional about how the volume should be consumed. For backwards
+compatibility, the absence of volumeType will default to volumes work today, which are formatted with a filesystem depending on 
+the plug-in chosen. Recycling will not be a supported reclaim policy. Once the user deletes the claim against a PV, the volume will 
+be scrubbed according to how it was bound. The path value in the local PV definition would be overloaded to define the path of the raw


emphasize that it's up to the plugin to scrub appropriately

msau42 · 2017-08-03T20:50:08Z

contributors/design-proposals/raw-block-pv.md

+
+## UC1: 
+
+DESCRIPTION: An admin wishes to pre-create a series of raw block devices to expose as PVs for consumption. The admin wishes to specify the purpose of these devices by specifying 'block' as the volumeType for the PVs.


Since you're using local volumes as the example, you can say "local raw block devices" in the description. Or, if you want it to be generic, and not just local, then maybe also give an example of some other plugins, like gce pd or fc.

@msau42 UC8 gives an example of GCE network attached raw block device. Updated UC1 to be local specific

msau42 · 2017-08-03T20:52:37Z

contributors/design-proposals/raw-block-pv.md

+```
+  NOTE: *accessModes correspond to the container runtime values. Where RWO == RWM (mknod) to enable the device to be written to and
+  create new files. (Default is RWM) ROX == R
+  **(RWX is NOT valid for block and should return an error.)** * This has been validated among runc, Docker and rocket. 


Is this something we can include in the PV validation?

@msau42 we don't do this today for any other parameters, it's just is for 'matching' labels. I am not against it for making usability better, but I think we would have to validate all access modes

Actually I think I am a little confused here. Access mode in this comment is referring to the container runtime values, and not the access mode that you specify in the PV/PVC.

So maybe this comment belongs in a section about CRI + runtimes.

@msau42 I think we are just noting this as a suggestion to the CRI team. That we have validated it for various runtimes and it should behave to respect those values.

Yeah, I think it would be valuable to have a section about CRI and how you plan on using CRI to implement this. Then this comment could go there. Right now it's kind of out of place in the user examples.

This note on access modes should be moved to the CRI section

msau42 · 2017-08-03T20:54:35Z

contributors/design-proposals/raw-block-pv.md

+
+***This has implementation details that have yet to be determined. It is included in this proposal for completeness of design ****
+
+## UC6: 


How is this different from UC3?

@msau42 UC6 uses a Storage Class and UC3 does not

msau42 · 2017-08-03T20:56:10Z

contributors/design-proposals/raw-block-pv.md

+metadata:
+  name: web
+spec:
+  serviceName: "nginx"


Yeah, so the example in UC7 shows a nginx application using ebs raw block volumes.

I think what you actually want, is an gluster application using ebs raw block volumes, and then an nginx application using gluster volumes.

msau42 · 2017-08-03T20:57:09Z

contributors/design-proposals/raw-block-pv.md

+  accessModes:
+    - "ReadWriteOnce"
+  gcePersistentDisk:
+    fsType: "raw"


Instead of special casing "raw" as a value for fsType, can fsType just be ignored/empty if volumeType == block?

jsafrane

I miss some technical details about how will we expose block devices to pods and how we make them usable by non-root processes.

jsafrane · 2017-08-04T13:50:04Z

contributors/design-proposals/raw-block-pv.md

+
+# Non Goals
+* Support all storage devices natively in upstream Kubernetes. Non-standard storage devices are expected to be managed using extension
+  mechanisms.


What extension mechanism? flex? CSI?

@jsafrane I don't know. It's a non-goal so we haven't figured that out yet and isn't included in this design.

For extension, flexvolume is a example.

@erinboyd Hypervisor based container runtime is one of best use cases here I think. xref https://github.com/kubernetes/frakti

thanks @resouer

jsafrane · 2017-08-04T13:58:52Z

contributors/design-proposals/raw-block-pv.md

+# Terminology
+* Raw Block Device - a physically attached device devoid of a filesystem
+* Raw Block Volume - a logical abstraction of the raw block device as defined by a path
+* File on Block - a formatted (ie xfs) filesystem on top of a raw block device


do you mean Filesystem on Block?

@jsafrane they will need to be exposed through the CRI (deviceinfo and devices) as part of the runtime interfaces.

I meant that File on Block does not make much sense, do you really mean single file on a block devices? Most people use filesystems for that.

+1 "Filesystem on block" is more clear.

jsafrane · 2017-08-04T13:59:27Z

contributors/design-proposals/raw-block-pv.md

+  and consumption of the devices. Since it's possible to create a volume as a block device and then later consume it by provisioning
+  a filesystem on top, the design requires explicit intent for how the volume will be used.
+  The additional benefit of explicitly defining how the volume is to be consumed will provide a means for indicating the method
+  by which the device should be scrubbed when the claim is deleted, as this method will differ from a raw block device compared to a 


Why should kubernetes / volume plugin scrub a device? Recycler is deprecated and local storage manages scrubbing in a different way, hidden from kubernetes.

Because if we go to reuse the volume, the data will still be there. Depending on the plugin type, the scrubbing may differ. For instance, if you are just using the raw block device vs. file on block, the clean up will be different. Therefore, it should fall to the plugin to 'know' how to scrub it.

I think the point you're trying to make here is that explicitly having volumeType in the PV and PVC can help the storage provider determine what scrubbing method to use. As an example, for local storage:

PV = block, PVC = block: zero the bytes

PV = block, PVC = file: destroy the filesystem

PV = file, PVC = file: delete the files

THanks @msau42 added this to the doc

Again, Kubernetes does not scrub devices. Recycler is deprecated, the only policies are Retain or Delete (the whole volume). So why is scrubbing part of this proposal? The only thing that scrubs devices can be an external DaemonSet for local storage. And that's outside of this PR, isn't it?

@jsafrane it will be plugin dependent on how to scrub the device. (Not that I am disagreeing with you above)

This sounds to me like another project which can be included later. I completely agree that scrubbing is important, but this is mainly on bare-metal deployments. On the cloud, when a network block device is deleted, the cloud vendor takes care of all of this.

@lpabon yes it is a different project and must be a consideration of each plugin. It is noted here for thoroughness

What is the difference between recycling and scrubbing? To be it's the same thing.

it will be plugin dependent on how to scrub the device

Plugins don't scrub anything. External local provisioner might want to do this, however it's external provisioner and we should not define the policy in Kubernetes.

+1 with @jsafrane- this is confusing. We do not want to imply that there is recycle or 'scrub' function of any sort encouraged by the API... You could say that the existing PV retention policies apply the same to PV.block as PV.file.. or not say anything.. but bringing up "scrub" implies recycle which is very deprecated.

jsafrane · 2017-08-04T14:50:49Z

contributors/design-proposals/raw-block-pv.md

+  by which the device should be scrubbed when the claim is deleted, as this method will differ from a raw block device compared to a 
+  filesystem. The ownership of scrubbing the device properly shall be up to the plugin method being utilized.
+  The last design point is block devices should be able to be fully restricted by the admin in accordance with how inline volumes 
+  are today. Ideally, the admin would want to be able to restrict either raw-local devices and or raw-network attached devices.


I wonder why would an admin want to restrict usage of block devices. What damage can an user do? The device is retained/deleted afterwards, i.e. nobody else can use it. The damage is IMO zero.

from @thockin in the google doc review: "I could easily write something that tastes like ext4 but causes the kernel
to do silly things like allocate infinite memory or dereference bad
pointers.

Mount is privileged for very good reasons."

Mount is privileged operation. If a pod is privileged, it can destroy the machine in many various ways, mounting a bad ext4 is quite complicated one. One does not need to use block device to mount a bad ext4, simple local file and mount -o loop is enough.

And there should be no way how to change a PVC with volumeType=block to volumeType=ext4, our validator should not allow this change (btw, a note in this proposal would be nice).

So, is there any other thing to be afraid of?

jsafrane · 2017-08-04T14:52:34Z

contributors/design-proposals/raw-block-pv.md

+name: myclaim
+spec:
+  volumeType: block #proposed API change
+accessModes:


indentation is wrong here, accesModes (and other fields) should be on the same level as volumeType

actually, most YAML examples in this PR have wrong indentation.

@jsafrane fixed

jsafrane · 2017-08-04T15:09:19Z

contributors/design-proposals/raw-block-pv.md

+
+ADMIN:
+
+* Admin creates a disk and attach it to server and pass-through the disk to KVM guest which Kubelet node1 is working inside.


the way I understand FC volume plugin is that it's the plugin who attaches a volume to a node like iSCSI or AWS does. There is no need for admin to attach the volume nor label nodes.

@jsafrane
In this case, kubelet is working inside a VM on hyper-visor. Therefore, admin have to create a disk and expose,attach the disk to the hyper-visor and also pass-through it to the VM beforehand.
(This might be done by external-provisioner if exits)

After that, FC plugin can find(attach) the volume to a container on the VM.

But then it's not a PV with FCVolumeSource, it looks like a HostPath or local volume. The thing that this volume is FC is irrelevant to the use case.

@jsafrane @erinboyd
Admin also have to create PV beforehand.
The second step mentioned that admin creates a PV using with device location(this will be defined to the FCVolumeSource). Is this still unclear?

@jsafrane
IIRC, following steps are needed to use pre-created FC volume.(note: these steps aren't virtual environmental case.)

Admin create a volume and expose the volume to a worker node.(This is done by storage operation)

Admin create FC PV using device location such as WWN and LUN.

User creates a PVC to bind pre-created FC PV.

User creates a Pod with PVC. During Pod creation, FC Plugin find FC disk which is attached to the worker node using "echo 1 > /sys...", then the FC volume is recognized to the worker node and the Pod consume this FC volume.

As you mentioned, I think FC volume is similar to a HostPath without external provisoner. But I think this is not a local volume case, because the FC volume is network attached volume and is not included inside the server local.

FC Plugin find FC disk which is attached to the worker node using "echo 1 > /sys...", then

Indeed, our FC plugin expects that someone attaches the volume before. I am sorry for confusion, I expected that our FC plugin is smarter and can attach a FC LUN by itself. So, could we use a simpler and more obvious use case with say iSCSI?

@jsafrane
Thank you for the confirmation.

I expected that our FC plugin is smarter and can attach a FC LUN by itself.

I see. Current FC plugin only has very limited functionality.

So, could we use a simpler and more obvious use case with say iSCSI?

How about following?

Admin creates a volume and expose it to kubelet node1 VM running on KVM.(This is done by storage operation)

Admin creates an iSCSI persistent volume using storage information such as portal IP, iqn and lun.

Admin adds "raw-disk" label to the kubelet node1 VM.

User creates a persistent volume claim with volumeType: block option to bind pre-created iSCSI PV.

User creates a Pod yaml which uses raw-pvc PVC and selects a node where kubelet node1 VM is running via nodeSelector option.
During Pod creation, iSCSI Plugin attaches iSCSI volume to the kubelet node1 VM using storage information.

Yes, that sounds better. Out of curiosity, why does the admin expose the volume just to one node? Wouldn't it be better if the volume could be used by all nodes in the cluster so the pod can run anywhere? That's IMO the typical use case.

@jsafrane
I forgot to change that scenario. That is specific to the FC plugin case. I'll change it.

As for the FC case, if we expose a FC volume to all worker nodes, the FC volume may be recognized by OS on those all worker nodes. If a container consumes the volume on a specific worker node, device file like /dev/sdX will be removed during volume deletion from the worker node, but the device file keeps remaining on other worker nodes even if it's unnecessary.
To avoid this situation, I suggested to expose FC volume to specific worker node and consume it on a selected worker node.

jsafrane · 2017-08-04T15:12:18Z

contributors/design-proposals/raw-block-pv.md

+      image: mysql
+      volumeMounts:
+      - name: my-db-data
+	mountPath: /var/lib/mysql/data


is /var/lib/mysql/data the right path for a device?

@jsafrane The PV would have the device path. This is the path for the container application

So, how is a raw block device exposed to a pod if it is not mountPath?

@jsafrane In the PV where it's defined.

PV defines source of the volume. How it is exposed to the pod? Above you wrote that it's not mountPath. So how is a MySQL in a container supposed to find the device?

@jsafrane The pod would specify the PVC in it, which is bound to the PV you created with the device path

@jsafrane from your comment below (I can't seem to comment on it). Why would you have several raw block devices exposed in a pod? I guess I thought we were trying to follow the same process for consuming raw block devices as we do filesystems and one PV would be == to one device. Therefore, a pod would only be able to use the single claim to a single rbd.

One pod can use several PersistentVolumes as filesystems right now. With raw block volumes I expect the same functionality. Simple example would be a Gluster brick, managing several physical hard drives.

@jsafrane personally I have never done this but I can understand the use case. I don't understand how it would be different then for this design as we are using the name conventions we use today for non-block devices by making a single PV to a single path..and in this case a device

The answer to @jsafrane question is the volumeMounts.mountPath will also be used.

jsafrane · 2017-08-04T15:18:24Z

contributors/design-proposals/raw-block-pv.md

+  accessModes:
+    - "ReadWriteOnce"
+  gcePersistentDisk:
+    fsType: "raw"


+1 to @msau42, if someone wants a raw device then volume plugin should ignore fsType part of the PV. If some crazy pod wants a raw device with a filesystem on it the pod can call mkfs on its own.

jsafrane · 2017-08-04T15:21:09Z

contributors/design-proposals/raw-block-pv.md

+    pdName: "gce-disk-1"
+```
+
+***If admin specifies volumeType: block + fstype: ext4 then they would get what they already get today ***


I don't understand this statement. Who gets what?

I think the statement is trying to say that "file on block" is today's behavior.

@jsafrane I made this more concise

jsafrane · 2017-08-04T15:26:41Z

contributors/design-proposals/raw-block-pv.md

+	  storage: 10Gi
+```
+***SUITABLE FOR: NETWORK ATTACHED BLOCK***
+


I would perhaps add an explicit use case where admin / dynamic provisioner creates a PV with volumeType: block and claim wants volumeType: file and everything works as usual - the PVC gets bound, the device is automatically formatted by kubernetes and mounted as a filesystem wherever a pod wants it.

So one PV can be consumed both as block or filesystem, depending on the PVC.

I would also add an example where claim wants volumeType: block, and the provisioner cannot support it.

@jsafrane @msau42 please look at the binding matrix to see if this is consistent

@jsafrane I was confused on that model also. I cannot come up with a use case where the PV is block and the claim is filesystem, just seem as that is what is done today.

@lpabon matrix updated...

ianchakeres · 2017-08-06T03:11:24Z

contributors/design-proposals/raw-block-pv.md

+
+  By extending the API for volumes to specifically request a raw block device, we provide an explicit method for volume comsumption,
+  whereas previously it was always a fileystem. In addition, the ability to use a raw block device without a filesystem will allow
+  Kuberenets better support of high performance applications that can utilitze raw block devices directly for their storage. 


/Kuberenets/Kubernetes/

ianchakeres · 2017-08-06T03:14:02Z

contributors/design-proposals/raw-block-pv.md

+  By extending the API for volumes to specifically request a raw block device, we provide an explicit method for volume comsumption,
+  whereas previously it was always a fileystem. In addition, the ability to use a raw block device without a filesystem will allow
+  Kuberenets better support of high performance applications that can utilitze raw block devices directly for their storage. 
+  For example, MariaDB or MongoDB.


Ceph Bluestore

ianchakeres · 2017-08-06T03:15:29Z

contributors/design-proposals/raw-block-pv.md

+  Therefore, the risk outweighed the reward when trying to  leverage the power of local storage needed for stateful applications like 
+  databases.
+
+  By extending the API for volumes to specifically request a raw block device, we provide an explicit method for volume comsumption,


/raw block device/raw block volume/

I am consistently using raw block device rather than volume as for some people volume implies filesystem and these are raw

The value add to kubernetes section seems incomplete. The last sentence seems like the only actual use case (i want to bypass the filesystem for better performance for a few databases), the rest seems like a background section.

This section should clearly describe why someone who has kubernetes as is today wants raw block devices - make the case that kubernetes should be changed. Give examples of use cases that raw block devices enable that are not possible today.

@smarterclayton The use cases below the value add section are all things you cannot do today in Kubernetes. I will notate this in this section to better develop the case for changing it.

ianchakeres · 2017-08-06T03:16:53Z

contributors/design-proposals/raw-block-pv.md

+  definitions for the block devices will be driven through the PVC and PV and Storage Class definitions. Along with Storage
+  Resource definitions, this will provide the admin with a consistent way of managing all storage. 
+  The API changes proposed in the following section are minimal with the idea of defining a volumeType to indicate both the definition
+  and consumption of the devices. Since it's possible to create a volume as a block device and then later consume it by provisioning


/of the devices/of the volumes/

jsafrane · 2017-08-07T11:36:38Z

contributors/design-proposals/raw-block-pv.md

+ 	GetVolumePath() string
+}
+```
+# Mounter binding matrix


PV binder does not have (and should not have) access to fsType in PV *VolumeSource. The whole table makes very little sense to me. Again, why do we need fsType=raw? It just complicates things unnecessarily.

@jsafrane you could have a provisioner that does both. Hence the provisioner takes the fsType in as a parameter. For instance, in GlusterFS they several provisioners that use the same base code but based on a flag it either installs a filesystem (fstype=ext4) onto the defvice or (fstype=raw) uses it as a raw block device. I just think if you want to automate something like Gluster E2E then you would need to have the ability to create or discover the block devices and then install the filesystem on it, instead of it being a multi-step process. Otherwise, it's not an improvement over what we have today which is very manual and error prone.

Again, binder does not see fsType and thus can't do any decision based on it. These two lines are equal to the binder, yet you assume different action.

PV volumeType Plugin fstype PVC volumeType Result

unspecified ext4 block NO BIND

unspecified raw block BIND

So are these lines (note that I added the second one as it's not covered by the table).

PV volumeType Plugin fstype PVC volumeType Result

unspecified raw unspecified NO BIND

unspecified ext4 unspecified BIND

For instance, in GlusterFS they several provisioners that use the same base code/

That's what PVC.VolumeType is for. Provisioner sees a PVC and provides the right PV (Gluster volume or iSCSI LUN) for it.

based on a flag it either installs a filesystem (fstype=ext4) onto the defvice or (fstype=raw) uses it as a raw block device.

Provisioners typically don't install filesystems on a device, that happens when the volume is mounted for the first time. And it won't have any FS created by kubernetes or provisioner when it's used as a raw block device.

you would need to have the ability to create or discover the block devices and then install the filesystem on it,

Kubernetes installs a FS on volumes when needed, why should a test do that?

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

@msau42

Automatic merge from submit-queue (batch tested with PRs 55112, 56029, 55740, 56095, 55845). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Block volume: Command line printer update **What this PR does / why we need it**: Add cmdline printer support changes. **Which issue this PR fixes**: Based on this proposal (kubernetes/community#805 & kubernetes/community#1265) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer**: There are another PRs related to this functionality. (#50457) API Change (#53385) VolumeMode PV-PVC Binding change (#51494) Container runtime interface change, volumemanager changes, operationexecutor changes (#55112) Block volume: Command line printer update Plugins (#51493) Block volumes Support: FC plugin update (#54752) Block volumes Support: iSCSI plugin update **Release note**: ``` NONE ``` /sig storage /cc @msau42 @jsafrane @saad-ali @erinboyd @screeley44 @kubernetes/sig-storage-pr-reviews - Command results ``` ~/sample/storage/fc_loop/file % k get pv,pvc,pod NAME CAPACITY ACCESS MODES VOLUME MODE RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv/block-pv0001 1Gi RWO Block Retain Bound default/nginx-block-pvc01 slow 2m pv/file-pv0001 1Gi RWO Filesystem Retain Bound default/nginx-file-pvc01 slow 24s NAME STATUS VOLUME CAPACITY ACCESS MODES VOLUME MODE STORAGECLASS AGE pvc/nginx-block-pvc01 Bound block-pv0001 1Gi RWO Block slow 2m pvc/nginx-file-pvc01 Bound file-pv0001 1Gi RWO Filesystem slow 25s NAME READY STATUS RESTARTS AGE po/nginx-file-pod1 0/1 ContainerCreating 0 4s po/nginx-pod1 1/1 Running 0 2m ~/sample/storage/fc_loop/file % k get pv,pvc,pod NAME CAPACITY ACCESS MODES VOLUME MODE RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pv/block-pv0001 1Gi RWO Block Retain Bound default/nginx-block-pvc01 slow 2m pv/file-pv0001 1Gi RWO Filesystem Retain Bound default/nginx-file-pvc01 slow 40s NAME STATUS VOLUME CAPACITY ACCESS MODES VOLUME MODE STORAGECLASS AGE pvc/nginx-block-pvc01 Bound block-pv0001 1Gi RWO Block slow 2m pvc/nginx-file-pvc01 Bound file-pv0001 1Gi RWO Filesystem slow 40s NAME READY STATUS RESTARTS AGE po/nginx-file-pod1 1/1 Running 0 19s po/nginx-pod1 1/1 Running 0 2m ~/sample/storage/fc_loop/file % k describe pv/block-pv0001 Name: block-pv0001 Labels: <none> Annotations: pv.kubernetes.io/bound-by-controller=yes volume.beta.kubernetes.io/storage-class=slow StorageClass: slow Status: Bound Claim: default/nginx-block-pvc01 Reclaim Policy: Retain Access Modes: RWO VolumeMode: Block Capacity: 1Gi Message: Source: Type: FC (a Fibre Channel disk) TargetWWNs: 28000001ff0414e2 LUN: 0 FSType: ReadOnly: true Events: <none> ~/sample/storage/fc_loop/file % k describe pv/file-pv0001 Name: file-pv0001 Labels: <none> Annotations: pv.kubernetes.io/bound-by-controller=yes volume.beta.kubernetes.io/storage-class=slow StorageClass: slow Status: Bound Claim: default/nginx-file-pvc01 Reclaim Policy: Retain Access Modes: RWO VolumeMode: Filesystem Capacity: 1Gi Message: Source: Type: FC (a Fibre Channel disk) TargetWWNs: 28000001ff0414e2 LUN: 0 FSType: ReadOnly: true Events: <none> ~/sample/storage/fc_loop/file % k describe pvc/nginx-block-pvc01 Name: nginx-block-pvc01 Namespace: default StorageClass: slow Status: Bound Volume: block-pv0001 Labels: <none> Annotations: pv.kubernetes.io/bind-completed=yes pv.kubernetes.io/bound-by-controller=yes volume.beta.kubernetes.io/storage-class=slow Capacity: 1Gi Access Modes: RWO VolumeMode: Block Events: <none> ~/sample/storage/fc_loop/file % k describe pvc/nginx-file-pvc01 Name: nginx-file-pvc01 Namespace: default StorageClass: slow Status: Bound Volume: file-pv0001 Labels: <none> Annotations: pv.kubernetes.io/bind-completed=yes pv.kubernetes.io/bound-by-controller=yes volume.beta.kubernetes.io/storage-class=slow Capacity: 1Gi Access Modes: RWO VolumeMode: Filesystem Events: <none> ```

@msau42

Automatic merge from submit-queue (batch tested with PRs 55938, 56055, 53385, 55796, 55922). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode binding logic update Adds VolumeMode binding logic to pv-controller for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** this change is dependent on #50457 cc @msau42 @jsafrane @mtanino @erinboyd

@msau42

…anager Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Block volumes Support: CRI, volumemanager and operationexecutor changes **What this PR does / why we need it**: This PR contains following items to enable block volumes support feature. - container runtime interface change - volumemanager changes - operationexecuto changes **Which issue this PR fixes**: Based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer**: There are another PRs related to this functionality. (#50457) API Change (#53385) VolumeMode PV-PVC Binding change (#51494) Container runtime interface change, volumemanager changes, operationexecutor changes (#55112) Block volume: Command line printer update Plugins (#51493) Block volumes Support: FC plugin update (#54752) Block volumes Support: iSCSI plugin update **Release note**: ``` Adds alpha support for block volume, which allows the users to attach raw block volume to their pod without filesystem on top of the volume. ``` /cc @msau42 @liggitt @jsafrane @saad-ali @erinboyd @screeley44

…rt-fc Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Block volumes Support: FC plugin update **What this PR does / why we need it**: Add interface changes to FC volume plugin to enable block volumes support feature. **Which issue this PR fixes**: Based on this proposal (kubernetes/community#805 & kubernetes/community#1265) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer**: This PR temporarily includes following changes except FC plugin change for reviewing purpose. These changes will be removed from the PR once they are merged. - (kubernetes#50457) API Change - (kubernetes#53385) VolumeMode PV-PVC Binding change - (kubernetes#51494) Container runtime interface change, volumemanager changes, operationexecutor changes There are another PRs related to this functionality. (kubernetes#50457) API Change (kubernetes#53385) VolumeMode PV-PVC Binding change (kubernetes#51494) Container runtime interface change, volumemanager changes, operationexecutor changes (kubernetes#55112) Block volume: Command line printer update Plugins (kubernetes#51493) Block volumes Support: FC plugin update (kubernetes#54752) Block volumes Support: iSCSI plugin update **Release note**: ``` FC plugin: Support for block volume - This enables uses to allow attaching raw block volume to their pod without filesystem through FC plugin. ```

k8s-github-robot · 2017-11-23T08:04:39Z

This PR hasn't been active in 30 days. It will be closed in 59 days (Jan 22, 2018).

cc @erinboyd @msau42

You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

@msau42

Automatic merge from submit-queue (batch tested with PRs 50457, 55558, 53483, 55731, 52842). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. VolumeMode and VolumeDevice api **What this PR does / why we need it:** Adds volumeType api to PV and PVC for local block support based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer:** There are other PR changes coming, this just simply creates the api fields #53385 - binding logic changes dependent on this change **Release note:** NONE Notes will be added in subsequents PR with the volume plugin changes, CRI, etc... cc @msau42 @liggitt @jsafrane @mtanino @saad-ali @erinboyd Kubernetes-commit: 5b32e4d24dd65573fc79b654c99f7c7f46de4ebc

erinboyd

This PR has broken GitHub and has been duplicated to #1265

Automatic merge from submit-queue (batch tested with PRs 54230, 58100, 57861, 54752). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Block volumes Support: iSCSI plugin update **What this PR does / why we need it**: Add interface changes to iSCSI volume plugin to enable block volumes support feature. **Which issue this PR fixes**: Based on this proposal (kubernetes/community#805 & kubernetes/community#1265) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer**: This PR temporarily includes following changes except iSCSI plugin change for reviewing purpose. These changes will be removed from the PR once they are merged. - (#50457) API Change - (#51494) Container runtime interface change, volumemanager changes, operationexecutor changes There are another PRs related to this functionality. (#50457) API Change (#53385) VolumeMode PV-PVC Binding change (#51494) Container runtime interface change, volumemanager changes, operationexecutor changes (#55112) Block volume: Command line printer update Plugins (#51493) Block volumes Support: FC plugin update (#54752) Block volumes Support: iSCSI plugin update **Release note**: ``` NONE ```

@msau42

…anager Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Block volumes Support: CRI, volumemanager and operationexecutor changes **What this PR does / why we need it**: This PR contains following items to enable block volumes support feature. - container runtime interface change - volumemanager changes - operationexecuto changes **Which issue this PR fixes**: Based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer**: There are another PRs related to this functionality. (#50457) API Change (#53385) VolumeMode PV-PVC Binding change (#51494) Container runtime interface change, volumemanager changes, operationexecutor changes (#55112) Block volume: Command line printer update Plugins (#51493) Block volumes Support: FC plugin update (#54752) Block volumes Support: iSCSI plugin update **Release note**: ``` Adds alpha support for block volume, which allows the users to attach raw block volume to their pod without filesystem on top of the volume. ``` /cc @msau42 @liggitt @jsafrane @saad-ali @erinboyd @screeley44

@msau42

…anager Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Block volumes Support: CRI, volumemanager and operationexecutor changes **What this PR does / why we need it**: This PR contains following items to enable block volumes support feature. - container runtime interface change - volumemanager changes - operationexecuto changes **Which issue this PR fixes**: Based on this proposal (kubernetes/community#805) and this feature issue: kubernetes/enhancements#351 **Special notes for your reviewer**: There are another PRs related to this functionality. (#50457) API Change (#53385) VolumeMode PV-PVC Binding change (#51494) Container runtime interface change, volumemanager changes, operationexecutor changes (#55112) Block volume: Command line printer update Plugins (#51493) Block volumes Support: FC plugin update (#54752) Block volumes Support: iSCSI plugin update **Release note**: ``` Adds alpha support for block volume, which allows the users to attach raw block volume to their pod without filesystem on top of the volume. ``` /cc @msau42 @liggitt @jsafrane @saad-ali @erinboyd @screeley44

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 12, 2017

erinboyd mentioned this pull request Jul 12, 2017

Raw block device using persistent volume source kubernetes/enhancements#351

Closed

fabiand reviewed Jul 17, 2017

View reviewed changes

ianchakeres reviewed Jul 25, 2017

View reviewed changes

mtanino reviewed Jul 25, 2017

View reviewed changes

screeley44 reviewed Jul 25, 2017

View reviewed changes

msau42 reviewed Jul 25, 2017

View reviewed changes

mtanino reviewed Jul 26, 2017

View reviewed changes

gnufied reviewed Jul 31, 2017

View reviewed changes

erinboyd changed the title ~~Raw block~~ Raw block volume support Aug 2, 2017

mtanino reviewed Aug 3, 2017

View reviewed changes

k8s-ci-robot requested a review from jsafrane August 3, 2017 17:00

k8s-ci-robot added the sig/storage Categorizes an issue or PR as relevant to SIG Storage. label Aug 3, 2017

msau42 reviewed Aug 3, 2017

View reviewed changes

jsafrane reviewed Aug 4, 2017

View reviewed changes

ianchakeres reviewed Aug 6, 2017

View reviewed changes

jsafrane reviewed Aug 7, 2017

View reviewed changes

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 23, 2017

erinboyd force-pushed the raw-block branch from 1c514ef to dce2cd4 Compare October 23, 2017 23:06

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 23, 2017

changes for design prop

4172654

erinboyd force-pushed the raw-block branch from dce2cd4 to 4172654 Compare October 24, 2017 04:32

erinboyd mentioned this pull request Oct 24, 2017

Version 2 for Raw Block Devices #1265

Merged

mtanino mentioned this pull request Oct 28, 2017

Block volumes Support: iSCSI plugin update kubernetes/kubernetes#54752

Merged

mtanino mentioned this pull request Nov 5, 2017

Block volume: Command line printer update kubernetes/kubernetes#55112

Merged

erinboyd mentioned this pull request Nov 16, 2017

[WIP] update aws-ebs to support raw block volumeMode kubernetes/kubernetes#55899

Closed

erinboyd commented Dec 8, 2017

View reviewed changes

k8s-github-robot added the kind/design Categorizes issue or PR as related to design. label Feb 6, 2018

erinboyd closed this Mar 5, 2018


		Authors: erinboyd@, screeley44@, mtanino@

		This document presents a proposal for managing raw block storage in Kubernetes using the persistent volume soruce API as a consistent model

		@@ -0,0 +1,571 @@
		# Local Raw Block Consumption via Persistent Volume Source


		# Value add to Kubernetes

		Before the advent of storage plugins, emptyDir and hostPath were widely used to quickly prototype stateless applications in Kube.


		DESCRIPTION:

		A developer wishes to enable their application to use a local raw block device as the volume for the container. The admin has already created PVs that the user will bind to by specifying 'block' as the volume type of their PVC.


		# Implementation Plan

		Phase 1: Pre-provisioned PVs to precreated devices

		@@ -0,0 +1,559 @@
		# Local Raw Block Consumption in Kubernetes


		## UC1:

		DESCRIPTION: An admin wishes to pre-create a series of raw block devices to expose as PVs for consumption. The admin wishes to specify the purpose of these devices by specifying 'block' as the volumeType for the PVs.


		*This has implementation details that have yet to be determined. It is included in this proposal for completeness of design **

		## UC6:


		ADMIN:

		* Admin creates a disk and attach it to server and pass-through the disk to KVM guest which Kubelet node1 is working inside.

PV volumeType	Plugin fstype	PVC volumeType	Result
unspecified	raw	unspecified	NO BIND
unspecified	ext4	unspecified	BIND

Raw block volume support #805

Raw block volume support #805

Conversation

erinboyd commented Jul 12, 2017

feiskyer commented Jul 13, 2017

fabiand left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erinboyd commented Jul 21, 2017 • edited Loading

erinboyd commented Jul 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtanino Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtanino commented Jul 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtanino Aug 4, 2017 • edited Loading

Choose a reason for hiding this comment

Mounter binding matrix for "Dynamic Provisioning"

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msau42 Jul 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtanino commented Jul 26, 2017

Choose a reason for hiding this comment

erinboyd commented Aug 3, 2017

erinboyd commented Jul 21, 2017 •

edited

Loading

mtanino Aug 1, 2017 •

edited

Loading

mtanino Aug 4, 2017 •

edited

Loading

msau42 Jul 25, 2017 •

edited

Loading