diff --git a/doc/source/test_plans/kubernetes_pod_recovery/code/Dockerfile b/doc/source/test_plans/kubernetes_pod_recovery/code/Dockerfile new file mode 100644 index 0000000..f84ee64 --- /dev/null +++ b/doc/source/test_plans/kubernetes_pod_recovery/code/Dockerfile @@ -0,0 +1,7 @@ +FROM debian:latest +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && apt-get -y upgrade +RUN apt-get -y --no-install-recommends install python + +ENTRYPOINT ["python", "-m", "SimpleHTTPServer"] diff --git a/doc/source/test_plans/kubernetes_pod_recovery/code/svc-dpl.yaml b/doc/source/test_plans/kubernetes_pod_recovery/code/svc-dpl.yaml new file mode 100644 index 0000000..8d7cc0b --- /dev/null +++ b/doc/source/test_plans/kubernetes_pod_recovery/code/svc-dpl.yaml @@ -0,0 +1,20 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: svc +spec: + replicas: 1 + template: + metadata: + labels: + app: svc + annotations: + scheduler.alpha.kubernetes.io/affinity: '{"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": + {"nodeSelectorTerms": [{"matchExpressions": [{"key": "kubernetes.io/hostname", + "operator": "In", "values": ["node4", "node5", "node6"]}]}]}}}' + spec: + containers: + - name: svc + image: 127.0.0.1:31500/qa/svc + ports: + - containerPort: 8000 diff --git a/doc/source/test_plans/kubernetes_pod_recovery/code/svc-svc.yaml b/doc/source/test_plans/kubernetes_pod_recovery/code/svc-svc.yaml new file mode 100644 index 0000000..a8e8d4c --- /dev/null +++ b/doc/source/test_plans/kubernetes_pod_recovery/code/svc-svc.yaml @@ -0,0 +1,13 @@ +apiVersion: v1 +kind: Service +metadata: + name: svc +spec: + ports: + - nodePort: 32333 + port: 8000 + protocol: TCP + targetPort: 8000 + selector: + app: svc + type: NodePort diff --git a/doc/source/test_plans/kubernetes_pod_recovery/plan.rst b/doc/source/test_plans/kubernetes_pod_recovery/plan.rst new file mode 100644 index 0000000..b664103 --- /dev/null +++ b/doc/source/test_plans/kubernetes_pod_recovery/plan.rst @@ -0,0 +1,80 @@ +.. _Kubernetes_pod_recovery_test_plan: + +********************************* +Kubernetes POD recovery test plan +********************************* + +:status: **ready** +:version: 1.0 + +:Abstract: + + This test plan covers scenarios for Kuberneter POD recovery testing. + +Test Plan +========= + +In Kubernetes "classic" HTTP-based services can be represented with help of +replication controller and service. Replication controller is responsible +for keeping number of PODs constant and in restarts POD in case of failure. +If the POD is deleted, then replication controller creates a new one. +Depending on scheduler settings a new instance of POD can be scheduled +to a different node. + +Under service performance we mean the amount of work produced by a service, +for HTTP service this can be measured as number of requests per second. + +Test Environment +---------------- + +Preparation +^^^^^^^^^^^ + +The test plan is executed against Kubernetes deployed on bare-metal nodes. + +Environment description +^^^^^^^^^^^^^^^^^^^^^^^ + +The environment description includes hardware specification of servers, +network parameters, operation system and OpenStack deployment characteristics. + + +Test Case #1: POD restart +------------------------- + +Description +^^^^^^^^^^^ + +In this test case we investigate how POD rescheduling affects service +performance. + +Steps: + * Build Docker image and push it into local registry: + ``docker build -t localhost:31500/qa/svc code/`` + ``docker push localhost:31500/qa/svc + * Create Kubernetes Service and Deployments based on files + :download:`code/svc-dpl.yaml` and :download:`code/svc-svc.yaml` + * Measure service performance during 20 seconds: + ``wrk -c 20 -d 10 -t 20 -H "Connection: close" --latency --timeout 10s http://svc.ccp:8000/`` + * While measurement is running delete the pod via + ``kubectl delete pod -l app=svc`` + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. table:: list of test metrics to be collected during this test + + +-------------------------+---------------------------------------------+ + | Parameter | Description | + +=========================+=============================================+ + | RPS_BASE | Base RPS value | + +-------------------------+---------------------------------------------+ + | RPS_WHILE_RESCHEDULING | RPS measured during rescheduling | + +-------------------------+---------------------------------------------+ + + +Reports +======= + +Test plan execution reports: + * :ref:`Kubernetes_pod_recovery_test_report` diff --git a/doc/source/test_results/container_cluster_systems/kubernetes/pod_recovery/chart.png b/doc/source/test_results/container_cluster_systems/kubernetes/pod_recovery/chart.png new file mode 100644 index 0000000..50f78e5 Binary files /dev/null and b/doc/source/test_results/container_cluster_systems/kubernetes/pod_recovery/chart.png differ diff --git a/doc/source/test_results/container_cluster_systems/kubernetes/pod_recovery/index.rst b/doc/source/test_results/container_cluster_systems/kubernetes/pod_recovery/index.rst new file mode 100644 index 0000000..3403a49 --- /dev/null +++ b/doc/source/test_results/container_cluster_systems/kubernetes/pod_recovery/index.rst @@ -0,0 +1,56 @@ +.. _Kubernetes_pod_recovery_test_report: + +****************************** +Kubernetes density test report +****************************** + +:Abstract: + + This document is the report for :ref:`Kubernetes_pod_recovery_test_plan` + + +Environment description +======================= + +This report is collected on the hardware described in +:ref:`intel_mirantis_performance_lab_1`. + + +Software +~~~~~~~~ + +Kubernetes is installed using :ref:`Kargo` deployment tool on Ubuntu 16.04.1. + +Node roles: + - node1: minion+master+etcd + - node2: minion+master+etcd + - node3: minion+etcd + - node4: minion + +Software versions: + - OS: Ubuntu 16.04.1 LTS (Xenial Xerus) + - Kernel: 4.4.0-47 + - Docker: 1.12.1 + - Kubernetes: 1.4.3 + +Reports +======= + +Test Case #1: POD restart +------------------------- + +With Kubernetes support of gracefully POD restart the service downtime is +minimized and may result in several requests to fail. The following chart is +based on 11 measurements: column 1 - no POD restart, 2 - with POD restart: + +.. image:: chart.png + +.. list-table:: POD stats + :header-rows: 1 + + * + - RPS_BASE + - RPS_WHILE_RESCHEDULING + * + - 1136 (stddev 162) + - 818 (stddev 158)