Kargo deployment performance
Change-Id: I28d1489e706a0d646432030e6a6e5a426ad96f59
This commit is contained in:
obasov
parent
f68133ee13
commit
2f552c7541
|
|
@ -14,9 +14,10 @@ Contents
|
|||
:numbered:
|
||||
:maxdepth: 2
|
||||
|
||||
kargo_deploy_performance
|
||||
performance_and_scaling
|
||||
API_latency
|
||||
|
||||
.. raw:: pdf
|
||||
|
||||
PageBreak oneColumn
|
||||
PageBreak oneColumn
|
||||
|
|
|
|||
|
|
@ -0,0 +1,265 @@
|
|||
|
||||
.. _Measuring_performance_of_Kargo:
|
||||
|
||||
=============================================
|
||||
Measuring performance of Kargo
|
||||
=============================================
|
||||
|
||||
:status: **ready**
|
||||
:version: 1.0
|
||||
|
||||
:Abstract:
|
||||
|
||||
This document describes a test plan for quantifying the performance of
|
||||
Kargo as a one of the Kubernetes deployment solutions. Kargo comes with
|
||||
Fuel Containerized Control Plane installer and intended to be fast,
|
||||
scalable and reliable K8S deployment tool. Its code mostly implemented
|
||||
as a set of ansible playbooks with some bash launcher scripts.
|
||||
|
||||
|
||||
Test Plan
|
||||
=========
|
||||
|
||||
This test plan aims to get overall performance, timing and resource
|
||||
utilization during K8S deployment with Kargo on a different number of
|
||||
nodes, using specified list of performance measurements and tools.
|
||||
|
||||
Test Environment
|
||||
----------------
|
||||
|
||||
Preparation
|
||||
^^^^^^^^^^^
|
||||
|
||||
1.
|
||||
The following package needs to be installed on the first node in the
|
||||
planned K8S cluster in order to collect performance metrics.
|
||||
|
||||
.. table:: Software to be installed
|
||||
|
||||
+--------------+---------+-----------------------------------+
|
||||
| package name | version | source |
|
||||
+==============+=========+===================================+
|
||||
| `dstat`_ | 0.7.2 | Ubuntu xenial universe repository |
|
||||
+--------------+---------+-----------------------------------+
|
||||
|
||||
Environment description
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Test results MUST include a description of the environment used. The following
|
||||
items should be included:
|
||||
|
||||
- **Hardware configuration of each server.** If virtual machines are used then
|
||||
both physical and virtual hardware should be fully documented.
|
||||
An example format is given below:
|
||||
|
||||
.. table:: Description of server hardware
|
||||
|
||||
+-------+----------------+-------+-------+
|
||||
|server |name | | |
|
||||
| +----------------+-------+-------+
|
||||
| |role | | |
|
||||
| +----------------+-------+-------+
|
||||
| |vendor,model | | |
|
||||
| +----------------+-------+-------+
|
||||
| |operating_system| | |
|
||||
+-------+----------------+-------+-------+
|
||||
|CPU |vendor,model | | |
|
||||
| +----------------+-------+-------+
|
||||
| |processor_count | | |
|
||||
| +----------------+-------+-------+
|
||||
| |core_count | | |
|
||||
| +----------------+-------+-------+
|
||||
| |frequency_MHz | | |
|
||||
+-------+----------------+-------+-------+
|
||||
|RAM |vendor,model | | |
|
||||
| +----------------+-------+-------+
|
||||
| |amount_MB | | |
|
||||
+-------+----------------+-------+-------+
|
||||
|NETWORK|interface_name | | |
|
||||
| +----------------+-------+-------+
|
||||
| |vendor,model | | |
|
||||
| +----------------+-------+-------+
|
||||
| |bandwidth | | |
|
||||
+-------+----------------+-------+-------+
|
||||
|STORAGE|dev_name | | |
|
||||
| +----------------+-------+-------+
|
||||
| |vendor,model | | |
|
||||
| +----------------+-------+-------+
|
||||
| |SSD/HDD | | |
|
||||
| +----------------+-------+-------+
|
||||
| |size | | |
|
||||
+-------+----------------+-------+-------+
|
||||
|
||||
- **Configuration of hardware network switches.** The configuration file from
|
||||
the switch can be downloaded and attached.
|
||||
|
||||
|
||||
- **Network scheme.** The plan should show how all hardware is connected and
|
||||
how the components communicate. All ethernet/fibrechannel and VLAN channels
|
||||
should be included. Each interface of every hardware component should be
|
||||
matched with the corresponding L2 channel and IP address.
|
||||
|
||||
Test Case
|
||||
---------
|
||||
|
||||
Description
|
||||
^^^^^^^^^^^
|
||||
|
||||
This specific test plan contains only one test case, that needs to be run
|
||||
step by step on the environments differing list of parameters below.
|
||||
|
||||
Parameters
|
||||
^^^^^^^^^^
|
||||
|
||||
=============== ===============
|
||||
Parameter name Value
|
||||
=============== ===============
|
||||
number of nodes 50, 150, 350
|
||||
=============== ===============
|
||||
|
||||
List of performance metrics
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The table below shows the list of test metrics to be collected. The priority
|
||||
is the relative ranking of the importance of each metric in evaluating the
|
||||
performance of the system.
|
||||
|
||||
.. table:: List of performance metrics
|
||||
|
||||
+--------+-------------------+-------------------+------------------------------------------+
|
||||
|Priority| Value | Measurement Units | Description |
|
||||
+========+===================+===================+==========================================+
|
||||
| | | || The elapsed time deploy ready K8S |
|
||||
| 1 | DEPLOYMENT_TIME | seconds || cluster on a different number of nodes. |
|
||||
| | | || |
|
||||
+--------+-------------------+-------------------+------------------------------------------+
|
||||
| | | || Total incoming/outgoing network |
|
||||
| 2 | NET_ALL | bit/s || bandwidth usage as a function of the |
|
||||
| | | || number of nodes. |
|
||||
+--------+-------------------+-------------------+------------------------------------------+
|
||||
| | | || CPU utilization as a function of the |
|
||||
| 3 | CPU | percentage || number of nodes. |
|
||||
+--------+-------------------+-------------------+------------------------------------------+
|
||||
| | | || Active memory usage as a function of |
|
||||
| 3 | RAM | MB || the number of nodes. |
|
||||
+--------+-------------------+-------------------+------------------------------------------+
|
||||
| | | || Storage total read/write IO bandwidth |
|
||||
| 3 | DISK_ALL | operations/second || as a function of the number of nodes. |
|
||||
+--------+-------------------+-------------------+------------------------------------------+
|
||||
|
||||
Measuring performance values
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
1.
|
||||
Start the collection of CPU, memory, network, and storage metrics on the
|
||||
first K8S cluster node. Use the dstat program which can collect all of
|
||||
these metrics in CSV format into a log file.
|
||||
2.
|
||||
Launch Kargo which will begin K8S deployment.
|
||||
3.
|
||||
Wait until the deployment process get finished.
|
||||
4.
|
||||
Stop the dstat program.
|
||||
5.
|
||||
Prepare collected data for analysis. dstat provides a large amount of
|
||||
information, which can be pruned by saving only the following:
|
||||
|
||||
* "system"[time]. Save as given.
|
||||
|
||||
* 100-"total cpu usage"[idl]. dstat provides only the idle CPU value. CPU
|
||||
utilization is calculated by subtracting the idle value from 100%.
|
||||
|
||||
* "memory usage"[used]. dstat provides this value in Bytes.
|
||||
This is converted it to Megabytes by dividing by 1024*1024=1048576.
|
||||
|
||||
* "net/total"[send/recv] The total receive and transmit bandwidth
|
||||
on the NIC. dstat provides these values in Bytes per second. They are
|
||||
converted to Bits per second dividing by 8.
|
||||
|
||||
* "io/total"[read]+"io/total"[writ]. The total read and write storage IO
|
||||
bandwidth.
|
||||
|
||||
These values will be graphed and maximum values reported.
|
||||
|
||||
Additional tests will be performed if some anomalous behavior is found.
|
||||
These may require the collection of additional performance metrics.
|
||||
|
||||
6.
|
||||
The result of this part of test will be:
|
||||
|
||||
* to provide the following graphs, one for each number of provisioned nodes:
|
||||
|
||||
#) One dependence on one graph.
|
||||
|
||||
* ALL_NET(TIME) Dependence on time of total network bandwidth usage.
|
||||
|
||||
#) One dependence on one graph.
|
||||
|
||||
* CPU(TIME) Dependence on time of CPU utilization.
|
||||
|
||||
#) One dependence on one graph.
|
||||
|
||||
* RAM(TIME) Dependence on time of active memory usage.
|
||||
|
||||
#) One dependence on one graph.
|
||||
|
||||
* ALL_IO(TIME) Dependence on time of total storage IO bandwidth.
|
||||
|
||||
|
||||
* to calculate following values and describe they in the table with dependency
|
||||
to the certain number of nodes:
|
||||
|
||||
.. table:: Maximum values of performance metrics
|
||||
|
||||
+-----------------------+-----+-----+-----+
|
||||
| number of nodes | 50 | 150 | 350 |
|
||||
+=======================+=====+=====+=====+
|
||||
| deployment time | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| cpu_usage_max | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| cpu_usage_min | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| cpu_usage_average | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| cpu_usage_percentile | | | |
|
||||
| 90% | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| ram_usage_max | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| ram_usage_min | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| ram_usage_average | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| ram_usage_percentile | | | |
|
||||
| 90% | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| net_all_max | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| net_all_min | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| net_all_average | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| net_all_percentile | | | |
|
||||
| 90% | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| dsk_io_all_max | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| dsk_io_all_min | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| dsk_io_all_average | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
| dsk_io_all_percentile | | | |
|
||||
| 90% | | | |
|
||||
+-----------------------+-----+-----+-----+
|
||||
|
||||
.. references:
|
||||
|
||||
.. _dstat: http://dag.wiee.rs/home-made/dstat/
|
||||
|
||||
|
||||
Reports
|
||||
=======
|
||||
|
||||
Test plan execution reports:
|
||||
* :ref:`Results_of_measuring_performance_of_Kargo`
|
||||
|
|
@ -7,3 +7,4 @@ Container cluster systems performance
|
|||
|
||||
mesos-marathon/mesos-marathon
|
||||
kubernetes/index
|
||||
kubernetes/deployment/index
|
||||
|
|
|
|||
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 200 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 133 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 118 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 122 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 201 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 131 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 110 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 107 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 195 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 152 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 132 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 257 KiB |
|
|
@ -0,0 +1,301 @@
|
|||
|
||||
.. _Results_of_measuring_performance_of_Kargo:
|
||||
|
||||
*******************************************
|
||||
Results of measuring performance of Kargo
|
||||
*******************************************
|
||||
|
||||
:Abstract:
|
||||
|
||||
This document includes performance test results of Kargo_
|
||||
as for Kubernetes deployment solution. All tests have been performed
|
||||
regarding :ref:`Measuring_performance_of_Kargo`.
|
||||
|
||||
Kargo sets up Kubernetes in the following way:
|
||||
|
||||
- master: Calico, Kubernetes API services
|
||||
- minion: Calico, Kubernetes minion services
|
||||
- etcd: etcd service
|
||||
|
||||
Kargo deploys Kubernetes cluster with the following matching hostnames and
|
||||
roles:
|
||||
|
||||
- node1: minion+master+etcd
|
||||
- node2: minion+master+etcd
|
||||
- node3: minion+etcd
|
||||
- all other nodes: minion
|
||||
|
||||
|
||||
Environment description
|
||||
=======================
|
||||
Hardware configuration of each server
|
||||
-------------------------------------
|
||||
|
||||
.. table:: Description of servers hardware
|
||||
|
||||
+-------+----------------+------------------------+------------------------+
|
||||
|server |name |node-{1..500} |node-{1..500} |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |role |kubernetes cluster | kubernetes cluster |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |vendor,model |Dell, R630 |Lenovo, RD550-1U |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |operating_system| | 4.4.0-36-generic | | 4.4.0-36-generic |
|
||||
| | | | Ubuntu-xenial | | Ubuntu-xenial |
|
||||
| | | | x86_64 | | x86_64 |
|
||||
+-------+----------------+------------------------+------------------------+
|
||||
|CPU |vendor,model |Intel, E5-2680v3 |Intel, E5-2680 v3 |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |processor_count |2 |2 |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |core_count |12 |12 |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |frequency_MHz |2500 |2500 |
|
||||
+-------+----------------+------------------------+------------------------+
|
||||
|RAM |vendor,model |Hynix HMA42GR7MFR4N-TF |Samsung M393A2G40DB0-CPB|
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |amount_MB |262144 |262144 |
|
||||
+-------+----------------+------------------------+------------------------+
|
||||
|NETWORK|interface_name |bond0 |bond0 |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |vendor,model |Intel, X710 Dual Port |Intel, X710 Dual Port |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |interfaces_count|2 |2 |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |bandwidth |10G |10G |
|
||||
+-------+----------------+------------------------+------------------------+
|
||||
|STORAGE|dev_name |/dev/sda |/dev/sda |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |vendor,model | | raid1 PERC H730P Mini| | raid1 MegaRAID 3108 |
|
||||
| | | | 2 disks Intel S3610 | | 2 disks Intel S3610 |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |SSD/HDD |SSD |SSD |
|
||||
| +----------------+------------------------+------------------------+
|
||||
| |size | 800GB | 800GB |
|
||||
+-------+----------------+------------------------+------------------------+
|
||||
|
||||
Network scheme and part of configuration of hardware network switches
|
||||
---------------------------------------------------------------------
|
||||
Network scheme of the environment:
|
||||
|
||||
.. image:: images/Network_Scheme.png
|
||||
:alt: Network Scheme of the environment
|
||||
:scale: 80
|
||||
|
||||
Here is the piece of switch configuration for each switch port which is a part of
|
||||
bond0 interface of a server:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
show run int et1
|
||||
interface Ethernet1
|
||||
description - r02r13c33
|
||||
switchport trunk native vlan 4
|
||||
switchport trunk allowed vlan 4
|
||||
switchport mode trunk
|
||||
channel-group 133 mode active
|
||||
lacp port-priority 16384
|
||||
spanning-tree portfast
|
||||
|
||||
show run int po1
|
||||
interface Port-Channel1
|
||||
description osscr02r13c21
|
||||
switchport trunk native vlan 131
|
||||
switchport trunk allowed vlan 130-159
|
||||
switchport mode trunk
|
||||
port-channel lacp fallback static
|
||||
port-channel lacp fallback timeout 30
|
||||
mlag 1
|
||||
|
||||
Software configuration of Kargo
|
||||
-----------------------------------------
|
||||
Setting up Kargo:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Kargo installation was performed on the bare metal Ubuntu Xenial servers. Kargo requires dedicated user (not root) to exist on the target nodes.
|
||||
To configure and launch Kargo section
|
||||
`Launcher script`_ has been used.
|
||||
|
||||
.. table:: Versions of some software
|
||||
|
||||
|
||||
+--------------------+-----------------------------------------+
|
||||
|Software |Version |
|
||||
+--------------------+-----------------------------------------+
|
||||
|Ubuntu |Ubuntu 16.04.1 LTS |
|
||||
+--------------------+-----------------------------------------+
|
||||
|fuel-ccp-installer |6b26170f70e523fb04bda8d6f15077d461fba9de |
|
||||
| | |
|
||||
+--------------------+-----------------------------------------+
|
||||
|kargo |016b7893c64fede07269c01cac31e96c8ee0d257 |
|
||||
+--------------------+-----------------------------------------+
|
||||
|
||||
Test tool:
|
||||
^^^^^^^^^^
|
||||
We were using `Dstat`_ utility as main tool for collecting timing and system performarce durring tests.
|
||||
`Script for parsing collected metrics`_ was used to parse performance metrics after installation tests.
|
||||
|
||||
Operating system configuration:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
You can find /etc folder contents from the one of the target servers where K8S cluster was deployed:
|
||||
:download:`etc_tarball_of_node1 <configs/node1.tar.gz>`
|
||||
|
||||
|
||||
Testing process
|
||||
===============
|
||||
|
||||
1.
|
||||
Kargo launcher script was set up and executed on node1 server as described in
|
||||
`Setting up Kargo:`_ section.
|
||||
|
||||
2.
|
||||
During Kargo run dstat tool was launched on the node1 with the following options:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
root@node1:~# dstat --nocolor --time --cpu --mem --net -N bond0 --io --output /root/dstat.csv
|
||||
|
||||
3.
|
||||
After finishing of Kargo run we parsed resulted "dstat.csv" files with `Script for parsing collected metrics`_.
|
||||
|
||||
|
||||
The above steps were repeated with the following numbers of nodes: 50,150,350
|
||||
|
||||
As a result of this part we got the following CSV files:
|
||||
|
||||
:download:`METRICS(NUMBER_OF_NODES=50) <./results/50_node1.csv>`
|
||||
|
||||
:download:`METRICS(NUMBER_OF_NODES=150) <./results/150_node1.csv>`
|
||||
|
||||
:download:`METRICS(NUMBER_OF_NODES=350) <./results/350_node1.csv>`
|
||||
|
||||
Results
|
||||
=======
|
||||
After simple processing results the following plots for performance metrics
|
||||
collected during provisioning of the nodes in depend on time created (click to
|
||||
expand an image):
|
||||
|
||||
CPU(TIME), RAM(TIME)
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|Number of|Plot CPU(TIME) |Plot RAM(TIME) |
|
||||
|nodes | | |
|
||||
+=========+=======================================+=======================================+
|
||||
|50 |.. image:: images/50_nodes_-_CPU.png |.. image:: images/50_nodes_-_RAM.png |
|
||||
| | :alt: CPU_USAGE(TIME, NODES=50) | :alt: RAM_USAGE(TIME, NODES=50) |
|
||||
| | :scale: 6 | :scale: 6 |
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|150 |.. image:: images/150_nodes_-_CPU.png |.. image:: images/150_nodes_-_RAM.png |
|
||||
| | :scale: 6 | :scale: 6 |
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|350 |.. image:: images/350_nodes_-_CPU.png |.. image:: images/350_nodes_-_RAM.png |
|
||||
| | :scale: 6 | :scale: 6 |
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|
||||
|
||||
NET(TIME), DISK(TIME)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|Number of|Plot NET(TIME) |Plot DISK(TIME) |
|
||||
|nodes | | |
|
||||
+=========+=======================================+=======================================+
|
||||
|50 |.. image:: images/50_nodes_-_net.png |.. image:: images/50_nodes_-_disk.png |
|
||||
| | :alt: CPU_USAGE(TIME, NODES=50) | :alt: RAM_USAGE(TIME, NODES=50) |
|
||||
| | :scale: 6 | :scale: 6 |
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|150 |.. image:: images/150_nodes_-_net.png |.. image:: images/150_nodes_-_disk.png |
|
||||
| | :scale: 6 | :scale: 6 |
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|350 |.. image:: images/350_nodes_-_net.png |.. image:: images/350_nodes_-_disk.png |
|
||||
| | :scale: 6 | :scale: 6 |
|
||||
+---------+---------------------------------------+---------------------------------------+
|
||||
|
||||
|
||||
The following table shows how performance metrics and deployment
|
||||
time parameters depend on a number of nodes.
|
||||
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|number of nodes |50 |150 |350 |
|
||||
+======================+===============+===============+===============+
|
||||
|deployment time |2049.00 |3922.00 |13065.00 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|cpu_usage_max |99.0210 |99.56 |99.06 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|cpu_usage_min |0 |0 |0 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|cpu_usage_average |7.2920 |10.03 |12.63 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|cpu_usage_percentile |19.6495 |24.92 |29.12 |
|
||||
|90% | | | |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|ram_usage_max |4466.10 |13859.56 |112079.57 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|ram_usage_min |1061.51 |1033.32 |1075.16 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|ram_usage_average |2121.20 |4335.69 |31288.94 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|ram_usage_percentile |2876.33 |8570.32 |79915.96 |
|
||||
|90% | | | |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|net_all_max |3864760.75 |20996615.75 |60130883.88 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|net_all_min |0 |0 |0 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|net_all_average |70602.55 |102913.32 |177943.40 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|net_all_percentile |253590.90 |263933.25 |180409.81 |
|
||||
|90% | | | |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|dsk_io_all_max |3503 |3196 |3470 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|dsk_io_all_min |0 |0 |0 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|dsk_io_all_average |26 |37 |56 |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|dsk_io_all_percentile |58 |14 |8 |
|
||||
|90% | | | |
|
||||
+----------------------+---------------+---------------+---------------+
|
||||
|
||||
Issues that have been found during the tests
|
||||
=============================================
|
||||
|
||||
During the testing we've found several issues that prevented us from
|
||||
achieving test results at scale:
|
||||
|
||||
+---------------------------------------+-----------------------------------------------+
|
||||
|Issue |Link |
|
||||
+=======================================+===============================================+
|
||||
|etcd list sometimes hangs | https://github.com/kubespray/kargo/pull/448 |
|
||||
+---------------------------------------+-----------------------------------------------+
|
||||
|K8S DNS services not working correctly | https://github.com/kubespray/kargo/pull/458 |
|
||||
+---------------------------------------+-----------------------------------------------+
|
||||
|Calico creates extra pool during run | https://github.com/kubespray/kargo/pull/462 |
|
||||
+---------------------------------------+-----------------------------------------------+
|
||||
|Timeout to quay.io to fetch etcd image | https://github.com/kubespray/kargo/pull/481 |
|
||||
+---------------------------------------+-----------------------------------------------+
|
||||
|Downloading images doesn't scale well | https://github.com/kubespray/kargo/pull/488 |
|
||||
+---------------------------------------+-----------------------------------------------+
|
||||
|Kargo is too slow on scale | https://github.com/kubespray/kargo/issues/478 |
|
||||
+---------------------------------------+-----------------------------------------------+
|
||||
|
||||
Applications
|
||||
============
|
||||
Launcher script
|
||||
------------------------------------
|
||||
.. literalinclude:: scripts/create_kargo_env.sh
|
||||
:language: bash
|
||||
|
||||
Script for parsing collected metrics
|
||||
------------------------------------------------
|
||||
|
||||
.. literalinclude:: scripts/parse_dstat.sh
|
||||
:language: bash
|
||||
|
||||
|
||||
.. references:
|
||||
|
||||
.. _Fuel-ccp-installer: https://github.com/openstack/fuel-ccp-installer
|
||||
.. _Kargo: https://github.com/kubespray/kargo
|
||||
.. _Dstat: http://dag.wiee.rs/home-made/dstat/
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,47 @@
|
|||
#!/bin/bash -xe
|
||||
|
||||
if [[ -d ./fuel-ccp-installer ]] ; then
|
||||
rm -rf ./fuel-ccp-installer
|
||||
fi
|
||||
|
||||
git clone https://review.openstack.org/openstack/fuel-ccp-installer
|
||||
|
||||
export ENV_NAME="kargo-test"
|
||||
export DEPLOY_METHOD="kargo"
|
||||
export WORKSPACE="~/workspace"
|
||||
export ADMIN_USER="vagrant"
|
||||
export ADMIN_PASSWORD="kargo"
|
||||
|
||||
# for 50 nodes
|
||||
#export SLAVES_COUNT=50
|
||||
#export ADMIN_IP="10.3.58.66"
|
||||
#export SLAVE_IPS="10.3.58.66 10.3.58.33 10.3.58.30 10.3.58.27 10.3.58.32 10.3.58.28 10.3.58.34 10.3.58.35 10.3.58.29 10.3.58.31 10.3.58.51 10.3.58.41 10.3.58.43 10.3.58.53 10.3.58.45 10.3.58.54 10.3.58.55 10.3.58.38 10.3.58.40 10.3.58.48 10.3.58.42 10.3.58.46 10.3.58.36 10.3.58.37 10.3.58.52 10.3.58.50 10.3.58.39 10.3.58.10 10.3.58.58 10.3.58.7 10.3.57.254 10.3.58.4 10.3.57.255 10.3.58.1 10.3.58.3 10.3.58.57 10.3.58.23 10.3.58.13 10.3.58.12 10.3.58.21 10.3.58.5 10.3.58.22 10.3.58.9 10.3.58.24 10.3.58.15 10.3.58.19 10.3.58.16 10.3.56.6 10.3.56.7 10.3.56.83"
|
||||
|
||||
# for 150 nodes:
|
||||
#export SLAVES_COUNT=150
|
||||
#export ADMIN_IP="10.3.56.3"
|
||||
#export SLAVE_IPS="10.3.56.3 10.3.56.254 10.3.56.4 10.3.56.6 10.3.56.7 10.3.56.83 10.3.56.82 10.3.56.84 10.3.56.86 10.3.56.87 10.3.56.89 10.3.56.12 10.3.56.11 10.3.56.13 10.3.56.15 10.3.56.16 10.3.56.17 10.3.56.18 10.3.56.20 10.3.56.21 10.3.56.22 10.3.56.23 10.3.56.25 10.3.56.26 10.3.56.27 10.3.56.29 10.3.56.30 10.3.56.31 10.3.56.32 10.3.56.34 10.3.56.33 10.3.56.37 10.3.56.38 10.3.56.39 10.3.56.41 10.3.56.43 10.3.56.45 10.3.56.46 10.3.56.47 10.3.56.48 10.3.56.50 10.3.56.49 10.3.56.51 10.3.56.52 10.3.56.133 10.3.56.135 10.3.56.137 10.3.56.136 10.3.56.113 10.3.56.139 10.3.56.141 10.3.56.148 10.3.56.142 10.3.56.117 10.3.56.143 10.3.56.145 10.3.56.123 10.3.56.122 10.3.56.128 10.3.56.144 10.3.56.250 10.3.56.251 10.3.56.126 10.3.56.180 10.3.56.181 10.3.56.184 10.3.56.182 10.3.56.185 10.3.56.183 10.3.56.188 10.3.56.191 10.3.56.192 10.3.56.187 10.3.56.195 10.3.56.190 10.3.56.199 10.3.56.193 10.3.56.204 10.3.56.207 10.3.56.205 10.3.56.206 10.3.56.201 10.3.56.202 10.3.56.208 10.3.56.217 10.3.56.216 10.3.56.209 10.3.56.210 10.3.56.215 10.3.56.218 10.3.56.212 10.3.56.213 10.3.56.214 10.3.56.211 10.3.56.221 10.3.56.224 10.3.56.227 10.3.56.149 10.3.56.219 10.3.56.223 10.3.56.231 10.3.56.228 10.3.56.235 10.3.56.236 10.3.56.230 10.3.56.233 10.3.56.229 10.3.56.232 10.3.56.234 10.3.59.95 10.3.59.92 10.3.59.88 10.3.59.96 10.3.59.111 10.3.59.115 10.3.59.116 10.3.56.146 10.3.59.119 10.3.59.117 10.3.59.112 10.3.59.110 10.3.59.109 10.3.59.120 10.3.59.137 10.3.59.136 10.3.59.133 10.3.59.132 10.3.59.138 10.3.59.134 10.3.59.135 10.3.59.139 10.3.59.131 10.3.59.130 10.3.59.74 10.3.59.80 10.3.59.73 10.3.59.77 10.3.59.84 10.3.59.105 10.3.59.82 10.3.59.83 10.3.59.81 10.3.59.98 10.3.59.108 10.3.59.106 10.3.59.102 10.3.59.107 10.3.59.86 10.3.58.66 10.3.58.33"
|
||||
|
||||
# for 350 nodes:
|
||||
#export SLAVES_COUNT=350
|
||||
#export ADMIN_IP="10.3.56.3"
|
||||
#export SLAVE_IPS="10.3.56.3 10.3.56.254 10.3.56.4 10.3.56.6 10.3.56.7 10.3.56.83 10.3.56.82 10.3.56.84 10.3.56.86 10.3.56.87 10.3.56.89 10.3.56.12 10.3.56.11 10.3.56.13 10.3.56.15 10.3.56.16 10.3.56.17 10.3.56.18 10.3.56.20 10.3.56.21 10.3.56.22 10.3.56.23 10.3.56.25 10.3.56.26 10.3.56.27 10.3.56.29 10.3.56.30 10.3.56.31 10.3.56.32 10.3.56.34 10.3.56.33 10.3.56.37 10.3.56.38 10.3.56.39 10.3.56.41 10.3.56.43 10.3.56.45 10.3.56.46 10.3.56.47 10.3.56.48 10.3.56.50 10.3.56.49 10.3.56.51 10.3.56.52 10.3.56.133 10.3.56.135 10.3.56.137 10.3.56.136 10.3.56.113 10.3.56.139 10.3.56.141 10.3.56.148 10.3.56.142 10.3.56.117 10.3.56.143 10.3.56.145 10.3.56.123 10.3.56.122 10.3.56.128 10.3.56.144 10.3.56.250 10.3.56.251 10.3.56.126 10.3.56.180 10.3.56.181 10.3.56.184 10.3.56.182 10.3.56.185 10.3.56.183 10.3.56.188 10.3.56.191 10.3.56.192 10.3.56.187 10.3.56.195 10.3.56.190 10.3.56.199 10.3.56.193 10.3.56.204 10.3.56.207 10.3.56.205 10.3.56.206 10.3.56.201 10.3.56.202 10.3.56.208 10.3.56.217 10.3.56.216 10.3.56.209 10.3.56.210 10.3.56.215 10.3.56.218 10.3.56.212 10.3.56.213 10.3.56.214 10.3.56.211 10.3.56.221 10.3.56.224 10.3.56.227 10.3.56.149 10.3.56.219 10.3.56.223 10.3.56.231 10.3.56.228 10.3.56.235 10.3.56.236 10.3.56.230 10.3.56.233 10.3.56.229 10.3.56.232 10.3.56.234 10.3.59.95 10.3.59.92 10.3.59.88 10.3.59.96 10.3.59.111 10.3.59.115 10.3.59.116 10.3.56.146 10.3.59.119 10.3.59.117 10.3.59.112 10.3.59.110 10.3.59.109 10.3.59.120 10.3.59.137 10.3.59.136 10.3.59.133 10.3.59.132 10.3.59.138 10.3.59.134 10.3.59.135 10.3.59.139 10.3.59.131 10.3.59.130 10.3.59.74 10.3.59.80 10.3.59.73 10.3.59.77 10.3.59.84 10.3.59.105 10.3.59.82 10.3.59.83 10.3.59.81 10.3.59.98 10.3.59.108 10.3.59.106 10.3.59.102 10.3.59.107 10.3.59.86 10.3.59.93 10.3.59.100 10.3.59.87 10.3.59.99 10.3.59.97 10.3.59.89 10.3.59.46 10.3.59.35 10.3.59.40 10.3.59.47 10.3.59.55 10.3.59.51 10.3.59.48 10.3.59.63 10.3.59.56 10.3.59.68 10.3.59.32 10.3.59.43 10.3.59.36 10.3.59.54 10.3.59.53 10.3.59.71 10.3.59.57 10.3.59.62 10.3.59.69 10.3.59.65 10.3.59.70 10.3.59.72 10.3.59.66 10.3.59.76 10.3.59.75 10.3.59.79 10.3.59.78 10.3.59.64 10.3.59.25 10.3.59.22 10.3.59.16 10.3.59.24 10.3.59.15 10.3.59.11 10.3.59.10 10.3.58.241 10.3.59.12 10.3.59.42 10.3.59.31 10.3.59.28 10.3.59.34 10.3.59.37 10.3.59.27 10.3.59.30 10.3.59.29 10.3.59.58 10.3.59.52 10.3.59.38 10.3.59.61 10.3.59.59 10.3.59.49 10.3.59.39 10.3.58.176 10.3.58.178 10.3.58.251 10.3.58.179 10.3.58.188 10.3.58.184 10.3.58.181 10.3.58.194 10.3.58.196 10.3.58.205 10.3.58.201 10.3.58.192 10.3.58.197 10.3.58.193 10.3.58.254 10.3.58.186 10.3.58.180 10.3.58.198 10.3.58.252 10.3.58.189 10.3.58.253 10.3.58.195 10.3.58.200 10.3.58.210 10.3.58.183 10.3.58.199 10.3.58.182 10.3.58.208 10.3.58.209 10.3.58.100 10.3.58.127 10.3.58.146 10.3.58.136 10.3.58.118 10.3.58.132 10.3.58.142 10.3.58.131 10.3.58.144 10.3.58.121 10.3.58.123 10.3.58.134 10.3.58.120 10.3.58.129 10.3.58.135 10.3.58.137 10.3.58.117 10.3.58.125 10.3.58.155 10.3.58.162 10.3.58.154 10.3.58.153 10.3.58.148 10.3.58.159 10.3.58.171 10.3.58.167 10.3.58.166 10.3.58.165 10.3.58.164 10.3.58.156 10.3.58.147 10.3.58.170 10.3.58.149 10.3.58.168 10.3.58.160 10.3.58.172 10.3.58.157 10.3.58.71 10.3.58.59 10.3.58.70 10.3.58.67 10.3.58.69 10.3.58.79 10.3.58.64 10.3.58.73 10.3.58.77 10.3.58.65 10.3.58.86 10.3.58.63 10.3.58.80 10.3.58.75 10.3.58.62 10.3.58.84 10.3.58.74 10.3.58.76 10.3.58.85 10.3.58.78 10.3.58.60 10.3.58.72 10.3.58.81 10.3.58.61 10.3.58.82 10.3.58.87 10.3.58.66 10.3.58.33 10.3.58.30 10.3.58.27 10.3.58.32 10.3.58.28 10.3.58.34 10.3.58.35 10.3.58.29 10.3.58.31 10.3.58.51 10.3.58.41 10.3.58.43 10.3.58.53 10.3.58.45 10.3.58.54 10.3.58.55 10.3.58.38 10.3.58.40 10.3.58.48 10.3.58.42 10.3.58.46 10.3.58.36 10.3.58.37 10.3.58.52 10.3.58.50 10.3.58.39 10.3.58.10 10.3.58.58 10.3.58.7 10.3.57.254 10.3.58.4 10.3.57.255 10.3.58.1 10.3.58.3 10.3.58.57 10.3.58.23 10.3.58.13 10.3.58.12 10.3.58.21 10.3.58.5 10.3.58.22 10.3.58.9 10.3.58.24 10.3.58.15 10.3.58.19 10.3.58.16"
|
||||
|
||||
export CUSTOM_YAML='docker_version: 1.12
|
||||
hyperkube_image_repo: "quay.io/coreos/hyperkube"
|
||||
hyperkube_image_tag: "v1.3.5_coreos.0"
|
||||
etcd_image_repo: "quay.io/coreos/etcd"
|
||||
etcd_image_tag: "v3.0.1"
|
||||
calicoctl_image_repo: "calico/ctl"
|
||||
#calico_node_image_repo: "calico/node"
|
||||
calico_node_image_repo: "l23network/node"
|
||||
calico_node_image_tag: "v0.20.0"
|
||||
calicoctl_image_tag: "v0.20.0"
|
||||
kube_apiserver_insecure_bind_address: "0.0.0.0"'
|
||||
|
||||
mkdir -p $WORKSPACE
|
||||
echo "Running on $NODE_NAME: $ENV_NAME"
|
||||
cd ./fuel-ccp-installer
|
||||
|
||||
bash -xe "./utils/jenkins/run_k8s_deploy_test.sh"
|
||||
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
#!/bin/bash -e
|
||||
|
||||
if [[ ! $1 ]] || [[ ! $2 ]] ; then
|
||||
echo \$1 = kargo_env_name, \$2 = csv file path
|
||||
exit 1
|
||||
fi
|
||||
|
||||
WORKDIR='~/worked_up_results/'
|
||||
cur_dir="${WORKDIR}kargo_${1}"
|
||||
csv_name=`basename $2`
|
||||
if [[ ! -d $cur_dir ]] ; then mkdir -p $cur_dir ; fi
|
||||
|
||||
awk -F "," 'BEGIN {getline;getline;getline;getline;getline;getline;getline;
|
||||
print "time,cpu_usage,ram_usage,net_recv,net_send,net_all,dsk_io_read,dsk_io_writ,dsk_all"}
|
||||
{printf "%s,%0.3f,%0.3f,%0.3f,%0.3f,%0.3f,%d,%d,%d\n", $1,100-$4,$8/1048576,$12/8,$13/8,($12+$13)/8,$14,$15,$14+$15 }' $2 > $cur_dir/${csv_name}
|
||||
|
||||
5
tox.ini
5
tox.ini
|
|
@ -5,6 +5,7 @@ skipsdist = True
|
|||
|
||||
[testenv]
|
||||
usedevelop = True
|
||||
basepython = python2.7
|
||||
# --ignore-installed is added to workaround problem with pip 8.0 and argparse
|
||||
# https://github.com/pypa/pip/issues/3404 and
|
||||
# https://github.com/pypa/pip/issues/3384
|
||||
|
|
@ -13,7 +14,9 @@ setenv = VIRTUAL_ENV={envdir}
|
|||
LANG=en_US.UTF-8
|
||||
LANGUAGE=en_US:en
|
||||
LC_ALL=C
|
||||
deps = -r{toxinidir}/test-requirements.txt
|
||||
deps = -r{toxinidir}/requirements.txt
|
||||
-r{toxinidir}/test-requirements.txt
|
||||
|
||||
commands =
|
||||
python setup.py test --slowest --testr-args='{posargs}'
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue