Kubernetes rally scenarios

Kubernetes Rally test plugin

Rally plugin for testing kubernetes.

Getting started

First of all, you need to create rally env for Kubernetes. There are two main ways to communicate to Kubernetes cluster - specifying auth-token or certifications. Choose what is suitable for your case and use one of the following samples.

To create env using certifications, use spec samples/platforms/kubernetes/cert-spec.yaml:

rally env create --name kubernetes --spec samples/platforms/kubernetes/cert-spec.yaml

For using Kubernetes token authentication, you need to get API key and use samples/platforms/kubernetes/apikey-spec.yaml spec to create env:

rally env create --name kubernetes --spec samples/platforms/kubernetes/apikey-spec.yaml

For initialization Rally environment to communicate to existing Kubernetes cluster you can also use system environment variables instead of making specification json/yaml file. See the list of available options:

• As like regular kubernetes client (kubectl) Rally can read kubeconfig file. Call rally env create --name kubernetes-created --from-sys-env and Rally with check $HOME/.kube/config file to the available configuration. Also, you can specify KUBECONFIG variable with a path different to the default $HOME/.kube/config.

• Despite the fact that kubectl doesn't support specifying Kubernetes credentials via separated system environment variables per separate option (auth_url, api_key, etc) like other platforms support (OpenStack, Docker, etc), Rally team provides this way. Check existing@kubernetes plugin for the list of all available variables. Here is a simple example of this feature:

  # the URL to the Kubernetes host.
  export KUBERNETES_HOST="https://example.com:3030"
  #  a path to a file containing TLS certificate to use when connecting to the Kubernetes host.
  export KUBERNETES_CERT_AUTH="~/.kube/cert_auth_file"
  # client API key to use as token when connecting to the Kubernetes host.
  export KUBERNETES_API_KEY="foo"
  # client API key prefix to use in token when connecting to the Kubernetes host.
  export KUBERNETES_API_KEY_PREFIX="bar"
  
  # finally create a Rally environment
  rally env create --name my-kubernetes --from-sysenv

Check env availbility by the following command:

rally env check

Now, if env is OK, create task config (or use it from samples/scenarios/kubernetes/) and start it:

rally task start samples/scenarios/kubernetes/run-namespaced-pods.yaml

After task is over, you can build report and results for further analyze.

There are next contexts for kubernetes tests:

Context	Sample	Description
namespaces	kubernetes.namespaces: count: 3 with_serviceaccount: yes	Creates count number of namespaces and non-default service accounts with tokens if necessary.
kubernetes.namespaces	kubernetes.namespaces: count: 3 with_serviceaccount: yes	[DEPRECATED!] Creates count number of namespaces and non-default service accounts with tokens if necessary.
kubernetes.local_storageclass	kubernetes.local_storageclass: {}	Creates local storage class according kubernetes documentation.
kubernetes.cfg	kubernetes.cfg: sleep_time: 0.5 retries_total: 100500 prepoll_delay: 1	rally-plugins utility method for overriding rally kubernetes config opts.

There are the following tasks:

Task	Description
Kubernetes.create_and_delete_pod	Creates pod, wait until it won't be running, collect pod's phases info and delete the pod.
Kubernetes.create_and_delete_replication_controller	Creates rc with number of replicas, wait until it won't be running and delete it.
Kubernetes.create_scale_and_delete_replication_controller	Scale rc with number of replicas, wait until it won't be running and revert scale it.
Kubernetes.create_delete_namespace	Creates namespace with random name, wait until it won't be active and then delete it.
Kubernetes.list_namespaces	List cluster namespaces.
Kubernetes.create_and_delete_pod_with_emptydir_volume	Create pod with emptyDir volume, wait until it won't be running, optionally exec pod with check_cmd and delete pod then.
Kubernetes.create_and_delete_pod_with_secret_volume	Create pod with secret volume, wait until it won't be running, optionally exec pod with check_cmd and delete pod then.
Kubernetes.create_and_delete_pod_with_hostpath_volume	Create pod with hostPath volume, wait until it won't be running, optionally exec pod with check_cmd and delete pod then.
Kubernetes.create_and_delete_pod_with_local_persistent_volume	Create pv, create pvc, create pod with pvc bound, wait until it won't be running, optionally exec pod with check_cmd and delete pod, pvc and pv then.
Kubernetes.create_and_delete_pod_with_hostpath_volume	Create pod with configMap volume, wait until it won't be running, optionally exec pod with check_cmd and delete pod then.
Kubernetes.create_and_delete_replicaset	Create replicaset with number of replicas, wait for all replicas are ready and delete replicaset then.
Kubernetes.create_scale_and_delete_replicaset	Create replicaset with number of replicas, wait for all replicas are ready, scale with scale_replicas, scale revert and delete replicaset then.
Kubernetes.create_and_delete_deployment	Create deployment with number of replicas, wait for all replicas are ready and delete deployment then.
Kubernetes.create_rollout_and_delete_deployment	Create deployment with number of replicas, wait for all replicas are ready, rollout with some changes and delete deployment then.
Kubernetes.create_and_delete_statefulset	Create statefulset with number of replicas, wait for all replicas are ready and delete statefulset then.
Kubernetes.create_scale_and_delete_statefulset	Create statefulset with number of replicas, wait for all replicas are ready, scale with scale_replicas, scale revert and delete statefulset then.
Kubernetes.create_and_delete_job	Create job with "Never" restart policy, wait for successful status and delete it then.
Kubernetes.create_check_and_delete_daemonset	Create daemonset, check each node contains the daemonset pods and delete daemonset then.
Kubernetes.create_check_and_delete_pod _with_cluster_ip_service	Create pod and clusterIP service with selector on the pod, run job with appropriate/curl image and curl command for pod endpoints, wait until job won't successfully finished and delete all resources then.
Kubernetes.create_check_and_delete_pod _with_cluster_ip_service_custom_endpoints	Create pod and clusterIP service w/o selector but with custom endpoints, run job with appropriate/curl image and curl command for pod endpoints, wait until job won't successfully finished and delete all resources then. This scenario tests not only service, Endpoints resource too.
Kubernetes.create_check_and_delete_pod _with_node_port_service	Create pod and nodePort service with selector on the pod, request server ip with created node port and delete all resources then.

Consider each task separately.

Kubernetes.create_and_delete_pod

The task contains next args:

Argument	Type	Description
image	string	image used in pod's manifest
command	array	array of strings representing container command, default is None

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-pod.yaml

Kubernetes.create_and_delete_replication_controller

The task contains next args:

Argument	Type	Description
replicas	number	number of replicas in RC
image	string	image used in replica's manifests
status_wait	bool	wait for resource status or not
command	array	array of strings representing container command, default is None

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-replication-controller.yaml

Kubernetes.create_scale_and_delete_replication_controller

The task contains next args:

Argument	Type	Description
replicas	number	original number of replicas
scale_replicas	number	number of replicas to scale
image	number	replication controller image
status_wait	bool	wait for resource status or not
command	array	array of strings representing container command, default is None

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-scale-and-delete-replication-controller.yaml

Kubernetes.create_and_delete_namespace

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-namespace.yaml

Kubernetes.list_namespaces

The task has no args.

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/list-namespaces.yaml

Kubernetes.create_and_delete_pod_with_emptydir_volume

The task contains next args:

Argument	Type	Description
image	string	image used in pod's manifest
mount_path	string	path to mount volume in pod
check_cmd	array	array of strings, which represents check command to exec in pod
error_regexp	string	Raise error if check_cmd response matches this arg regexp
status_wait	bool	wait for resource status or not
command	array	array of strings representing container command, default is None

If check_cmd is specified in task config, check_volume_pod action will be called.

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-check-and-delete-emptydir-volume.yaml

Kubernetes.create_and_delete_pod_with_secret_volume

The task contains next args:

Argument	Type	Description
image	string	image used in pod's manifest
mount_path	string	path to mount volume in pod
check_cmd	array	array of strings, which represents check command to exec in pod
error_regexp	string	Raise error if check_cmd response matches this arg regexp
status_wait	bool	wait for resource status or not
command	array	array of strings representing container command, default is None

If check_cmd is specified in task config, check_volume_pod action will be called.

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-check-and-delete-secret-volume.yaml

Kubernetes.create_and_delete_pod_with_hostpath_volume

The task contains next args:

Argument	Type	Description
image	string	image used in pod's manifest
mount_path	string	path to mount volume in pod
volume_type	string	hostPath type according kubernetes api
volume_path	string	hostPath path to mount from host
check_cmd	array	array of strings, which represents check command to exec in pod
error_regexp	string	Raise error if check_cmd response matches this arg regexp
status_wait	bool	wait for resource status or not
command	array	array of strings representing container command, default is None

If check_cmd is specified in task config, check_volume_pod action will be called.

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-check-and-delete-hostpath-volume.yaml

Kubernetes.create_and_delete_pod_with_local_persistent_volume

The task contains next args:

Argument	Type	Description
persistent_volume	map	persistent volume valuable params
-> size	string	PV size in kubernetes size format
-> volume_mode	string	Filesystem or Block
-> local_path	string	PV local path to volume on host
-> access_modes	list	PV access modes list of strings
-> node_affinity	map	PV nodeAffinity rule
persistent_volume_claim	map	PVC valuable params
-> size	string	PVC size in kubernetes size format
-> access_modes	list	PVC access modes list of strings
image	string	image used in pod's manifest
mount_path	string	path to mount volume in pod
check_cmd	array	array of strings, which represents check command to exec in pod
error_regexp	string	Raise error if check_cmd response matches this arg regexp
status_wait	bool	wait for resource status or not
command	array	array of strings representing container command, default is None

If check_cmd is specified in task config, check_volume_pod action will be called.

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-pod-with-local-persistent-volume.yaml

Kubernetes.create_and_delete_pod_with_configmap_volume

The task contains next args:

Argument	Type	Description
image	string	image used in pod's manifest
mount_path	string	path to mount volume in pod
configmap_data	map	configMap resource data
subpath	string	subPath cm data to mount in pod
sleep_time	number	sleep time between each two retries
check_cmd	array	array of strings, which represents check command to exec in pod
error_regexp	string	Raise error if check_cmd response matches this arg regexp
status_wait	bool	wait for resource status or not
command	array	array of strings representing container command, default is None

If check_cmd is specified in task config, check_volume_pod action will be called.

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-pod-with-configmap-volume.yaml

Kubernetes.create_and_delete_replicaset

The task contains next args:

Argument	Type	Description
replicas	number	number of replicas in replicaset
image	string	image used in replica's manifests
name	string	replicaset custom name, default is random
command	array	array of strings representing container command, default is None
status_wait	bool	wait for status if True

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-replicaset.yaml

Kubernetes.create_scale_and_delete_replicaset

The task contains next args:

Argument	Type	Description
replicas	number	number of replicas in replicaset
scale_replicas	number	number of replicas to scale
image	string	image used in replica's manifests
name	string	replicaset custom name, default is random
command	array	array of strings representing container command, default is None
status_wait	bool	wait for status if True

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-scale-and-delete-replicaset.yaml

Kubernetes.create_and_delete_deployment

The task contains next args:

Argument	Type	Description
replicas	number	number of replicas in deployment
image	string	image used in replica's manifests
name	string	deployment custom name, default is random
command	array	array of strings representing container command, default is None
status_wait	bool	wait for status if True

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-deployment.yaml

Kubernetes.create_rollout_and_delete_deployment

The task contains next args:

Argument	Type	Description
replicas	number	number of replicas in deployment
env	array	array of mappings representing kubernetes container's env
resources	map	map representing container resources requirements
changes	map	map with allowed keys env, resources or image for rollout deployment
image	string	image used in replica's manifests
name	string	replicaset custom name, default is random
command	array	array of strings representing container command, default is None
status_wait	bool	wait for status if True

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-rollout-and-delete-deployment.yaml

Kubernetes.create_and_delete_statefulset

The task contains next args:

Argument	Type	Description
replicas	number	number of replicas in replicaset
image	string	image used in replica's manifests
name	string	replicaset custom name, default is random
command	array	array of strings representing container command, default is None
status_wait	bool	wait for status if True

The task supports rps and constant types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-statefulset.yaml

Kubernetes.create_scale_and_delete_statefulset

The task contains next args:

Argument	Type	Description
replicas	number	number of replicas in replicaset
scale_replicas	number	number of replicas to scale
image	string	image used in replica's manifests
name	string	replicaset custom name, default is random
command	array	array of strings representing container command, default is None
status_wait	bool	wait for status if True

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-scale-and-delete-statefulset.yaml

Kubernetes.create_and_delete_job

The task contains next args:

Argument	Type	Description
image	string	image used in container's spec
name	string	job custom name, default is random
command	array	array of strings representing container command
status_wait	bool	wait for succeeded job if True

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-and-delete-job.yaml

Kubernetes.create_check_and_delete_daemonset

The task contains next args:

Argument	Type	Description
image	string	image used in container's spec
name	string	daemonset custom name, default is random
node_labels	map	labels keypairs, with which node list will be filtered, i.e. on which nodes daemonset pods have been created.
command	array	array of strings representing container command
status_wait	bool	wait for all pods will be ready

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-check-and-delete-daemonset.yaml

Kubernetes.create_check_and_delete_pod_with_cluster_ip_service

The task contains next args:

Argument	Type	Description
image	string	image used in container's spec
port	integer	port used as container port and in service
protocol	integer	protocol used as container port protocol and in service
image	string	image used in container's spec
name	string	daemonset custom name, default is random
command	array	array of strings representing container command
status_wait	bool	wait for all pods will be ready

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-check-and-delete-pod-with-cluster-ip-service.yaml

Kubernetes.create_check_and_delete_pod_with_cluster_ip_service_custom_endpoints

The task contains next args:

Argument	Type	Description
image	string	image used in container's spec
port	integer	port used as container port and in service
protocol	integer	protocol used as container port protocol and in service
image	string	image used in container's spec
name	string	daemonset custom name, default is random
command	array	array of strings representing container command
status_wait	bool	wait for all pods will be ready

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-check-and-delete-pod-with-cluster-ip-service-custom-endpoints.yaml

Kubernetes.create_check_and_delete_pod_with_node_port_service

The task contains next args:

Argument	Type	Description
image	string	image used in container's spec
port	integer	port used as container port and in service
request_timeout	integer	GET request timeout for check nodePort service
protocol	integer	protocol used as container port protocol and in service
image	string	image used in container's spec
name	string	daemonset custom name, default is random
command	array	array of strings representing container command
status_wait	bool	wait for all pods will be ready

The task supports constant and rps types of scenario runner.

To run the test, run next command:

rally task start samples/scenarios/kubernetes/create-check-and-delete-pod-with-node-port-service.yaml