Add shaker test for cluster in upgrade

Add shaker test plans and test results for ccp cluster during upgrade
for Mitaka to Newton

Change-Id: I5e3fc83048dc28b00a5c10fc50067d2efaf09106
This commit is contained in:
Sergey 2017-04-17 20:02:01 +04:00 committed by Sergey Galkin
parent 509c5bf8ae
commit 18f7b6e7f7
12 changed files with 2109 additions and 7 deletions

View File

@ -17,13 +17,9 @@ Fuel Containerized Control Plane upgradability performance testing
Test Plan
=========
This document is inspired by
:ref:`openstack_control_plane_performance_test_plan`, and aims to cover
baseline cloud operations and extend this test suite to verify containerized
deployment approach. As :ref:`openstack_control_plane_performance_test_plan`
this test plan covers basic cloud operation VMs creation, as well as
long-running test suites to verify cloud stability during update Open
Stack from Mitaka to Newton.
This test plan covers basic network performance with long-running test suites
to verify cloud network stability and performance during update Open Stack
from Mitaka to Newton.
Test Environment
----------------

View File

@ -0,0 +1,189 @@
.. _neutron_fuel_ccp_upgrade_test_plans:
=============================================================
OpenStack Neutron Testing During Open Stack Upgrade Test Plan
=============================================================
:status: **ready**
:version: 1.0
:Abstract:
This test plan aims to provide set of tests to identify OpenStack
performance against given containerized OpenStack cloud (installed
on the top of pre-deployed Kubernetes cluster) using simple minimalistic set
of Shaker tests during upgrade from Mitaka to Newton.
:Conventions:
- **Shaker** - Data plane performance testing tool
- **iperf** - Commonly-used network testing tool
Test Plan
=========
The purpose of this section is to describe scenarios for testing Open Stack
networking during upgrade from Mitaka to Newton.
Test Environment
----------------
Preparation
^^^^^^^^^^^
This test plan is performed against existing OpenStack cloud installed on top
of pre-deployed Kubernetes cluster with `fuel-ccp`_ tool with pre-installed Rally
framework.
.. _fuel-ccp: http://fuel-ccp.readthedocs.io/en/latest/
Environment description
^^^^^^^^^^^^^^^^^^^^^^^
The environment description includes hardware specification of servers,
network parameters, operation system and OpenStack deployment characteristics.
Hardware
~~~~~~~~
This section contains list of all types of hardware nodes (table below is
an example).
+-----------+-------+----------------------------------------------------+
| Parameter | Value | Comments |
+-----------+-------+----------------------------------------------------+
| model | | e.g. Supermicro X9SRD-F |
+-----------+-------+----------------------------------------------------+
| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz |
+-----------+-------+----------------------------------------------------+
| role | | e.g. compute or network |
+-----------+-------+----------------------------------------------------+
Network
~~~~~~~
This section contains list of interfaces and network parameters. For
complicated cases this section may include topology diagram and switch
parameters (table below is an example).
+------------------+-------+-------------------------+
| Parameter | Value | Comments |
+------------------+-------+-------------------------+
| network role | | e.g. provider or public |
+------------------+-------+-------------------------+
| card model | | e.g. Intel |
+------------------+-------+-------------------------+
| driver | | e.g. ixgbe |
+------------------+-------+-------------------------+
| speed | | e.g. 10G or 1G |
+------------------+-------+-------------------------+
| MTU | | e.g. 9000 |
+------------------+-------+-------------------------+
| offloading modes | | e.g. default |
+------------------+-------+-------------------------+
Software
~~~~~~~~
This section describes installed software (table below is an example).
+-----------------+-------+---------------------------+
| Parameter | Value | Comments |
+-----------------+-------+---------------------------+
| OS | | e.g. Ubuntu 14.04.3 |
+-----------------+-------+---------------------------+
| OpenStack | | e.g. Mitaka |
+-----------------+-------+---------------------------+
| Hypervisor | | e.g. KVM |
+-----------------+-------+---------------------------+
| Neutron plugin | | e.g. ML2 + OVS |
+-----------------+-------+---------------------------+
| L2 segmentation | | e.g. VLAN / VxLAN / GRE |
+-----------------+-------+---------------------------+
| virtual routers | | e.g. HA / DVR |
+-----------------+-------+---------------------------+
Test tool
---------
Shaker wraps around popular system network testing tools like iperf, iperf3
and netperf (with help of flent). Shaker is able to deploy OpenStack instances
and networks in different topologies. Shaker scenario specifies the deployment
and list of tests to execute. Additionally tests may be tuned dynamically
in command-line.
Test Case 1: Analysis of L2 metrics during Open Stack upgrade
-------------------------------------------------------------
Description
^^^^^^^^^^^
`Shaker <http://pyshaker.readthedocs.org/en/latest/index.html>`__ is
able to deploy OpenStack instances and networks in different topologies.
The following steps should be executed:
1. Deploy Kubernetes cluster
2. Deploy Open Stack cluster
3. Run full_l2 shaker's scenario and wait successfully stack heats deployment
4. Start Open Stack upgrade
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. table:: Shaker metrics
======== =============== ================= ======================================
Priority Value Measurement Units Description
======== =============== ================= ======================================
1 Errors times Number of errors during testing
1 Lost times Number of lost results during testing
1 Latency ms The network latency
1 TCP bandwidth Mbits/s TCP network bandwidth
2 UDP bandwidth packets per sec Number of UDP packets of 32 bytes size
2 TCP retransmits packets per sec Number of retransmitted TCP packets
======== =============== ================= ======================================
Test Case 2: Analysis of L3 metrics during Open Stack upgrade
-------------------------------------------------------------
Description
^^^^^^^^^^^
`Shaker <http://pyshaker.readthedocs.org/en/latest/index.html>`__ is
able to deploy OpenStack instances and networks in different topologies.
The following steps should be executed:
1. Deploy Kubernetes cluster
2. Deploy Open Stack cluster
3. Run full_l3_east_west shaker's scenario and wait successfully stack heats
deployment
4. Start Open Stack upgrade
List of performance metrics
^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. table:: Shaker metrics
======== =============== ================= ======================================
Priority Value Measurement Units Description
======== =============== ================= ======================================
1 Errors times Number of errors during testing
1 Lost times Number of lost results during testing
1 Latency ms The network latency
1 TCP bandwidth Mbits/s TCP network bandwidth
2 UDP bandwidth packets per sec Number of UDP packets of 32 bytes size
2 TCP retransmits packets per sec Number of retransmitted TCP packets
======== =============== ================= ======================================
Reports
=======
Test plan execution reports:
* :ref:`neutron_fuel_ccp_upgrade_report`

Binary file not shown.

After

Width:  |  Height:  |  Size: 12 KiB

View File

@ -0,0 +1,17 @@
builder:
push: true
no_cache: false
registry:
address: "172.20.9.32:5000/env-1"
repositories:
skip_empty: True
kubernetes:
server: http://127.0.0.1:8080
image_pull_policy: Always
---
!include
- versions.yaml
- topology.yaml
- configs.yaml
- repos.yaml

View File

@ -0,0 +1,64 @@
configs:
private_interface: p1p1.602
public_interface: p1p1.602
ingress:
enabled: true
glance:
tls:
enabled: false
bootstrap:
enable: true
neutron:
tls:
enabled: false
physnets:
- name: "physnet1"
bridge_name: "br-external"
interface: "bond0.648"
flat: true
vlan_range: false
dpdk: false
bootstrap:
internal:
enable: true
external:
enable: true
net_name: ext-net
subnet_name: ext-subnet
physnet: physnet1
network: 172.20.172.0/22
gateway: 172.20.172.1
nameserver: 8.8.8.8
pool:
start: 172.20.173.0
end: 172.20.175.254
keystone:
debug: true
security:
tls:
create_certificates: false
etcd:
tls:
enabled: false
db:
tls:
enabled: false
rabbitmq:
tls:
enabled: false
keystone:
tls:
enabled: false
cinder:
tls:
enabled: false
heat:
tls:
enabled: false
nova:
tls:
enabled: false
horizon:
tls:
enabled: false

View File

@ -0,0 +1,39 @@
repositories:
repos:
- git_url: https://git.openstack.org/openstack/fuel-ccp-ceph
name: fuel-ccp-ceph
- git_url: https://git.openstack.org/openstack/fuel-ccp-debian-base
name: fuel-ccp-debian-base
- git_url: https://git.openstack.org/openstack/fuel-ccp-entrypoint
name: fuel-ccp-entrypoint
- git_url: https://git.openstack.org/openstack/fuel-ccp-etcd
name: fuel-ccp-etcd
- git_url: https://git.openstack.org/openstack/fuel-ccp-galera
name: fuel-ccp-galera
- git_url: https://git.openstack.org/openstack/fuel-ccp-glance
name: fuel-ccp-glance
- git_url: https://git.openstack.org/openstack/fuel-ccp-heat
name: fuel-ccp-heat
- git_url: https://git.openstack.org/openstack/fuel-ccp-horizon
name: fuel-ccp-horizon
- git_url: https://git.openstack.org/openstack/fuel-ccp-ironic
name: fuel-ccp-ironic
- git_url: https://git.openstack.org/openstack/fuel-ccp-keystone
name: fuel-ccp-keystone
- git_url: https://git.openstack.org/openstack/fuel-ccp-mariadb
name: fuel-ccp-mariadb
- git_url: https://git.openstack.org/openstack/fuel-ccp-memcached
name: fuel-ccp-memcached
- git_url: https://git.openstack.org/openstack/fuel-ccp-murano
name: fuel-ccp-murano
- git_url: https://git.openstack.org/openstack/fuel-ccp-neutron
name: fuel-ccp-neutron
- git_url: https://git.openstack.org/openstack/fuel-ccp-nginx
name: fuel-ccp-nginx
- git_url: https://git.openstack.org/openstack/fuel-ccp-nova
name: fuel-ccp-nova
- git_url: https://git.openstack.org/openstack/fuel-ccp-openstack-base
name: fuel-ccp-openstack-base
- git_url: https://git.openstack.org/openstack/fuel-ccp-rabbitmq
name: fuel-ccp-rabbitmq

View File

@ -0,0 +1,86 @@
nodes:
# node[1-3]: ETCD
# node[4-5]: Kubernetes master
node([6|7|8])$: # 6-8
roles:
- controller
- openvswitch
- messaging
- db
- indentity
node9$:
roles:
- etc
- backups
- indentity
node[1-9][0-9]$: # 10-99
roles:
- compute
- openvswitch
- indentity
replicas:
database: 3
etcd: 1
rpc: 3
notifications: 3
glance-api: 1
glance-registry: 1
keystone: 3
nova-api: 3
nova-scheduler: 3
nova-conductor: 3
neutron-server: 3
neutron-metadata-agent: 3
horizon: 3
heat-api: 3
heat-engine: 3
roles:
db:
- database
messaging:
- rpc
- notifications
controller:
- etcd
- glance-api
- glance-registry
- heat-api-cfn
- heat-api
- heat-engine
- horizon
- memcached
- neutron-dhcp-agent
- neutron-l3-agent
- neutron-metadata-agent
- neutron-server
- nova-api
- nova-conductor
- nova-consoleauth
- nova-novncproxy
- nova-scheduler
compute:
- nova-compute
- nova-libvirt
openvswitch:
- neutron-openvswitch-agent
- openvswitch-db
- openvswitch-vswitchd
etc:
- etcd
indentity:
- keystone
backups:
- backup
services:
database:
service_def: galera
rpc:
service_def: rabbitmq
notifications:
service_def: rabbitmq

View File

@ -0,0 +1,41 @@
images:
tag: mitaka
sources:
openstack/cinder:
git_ref: stable/mitaka
git_url: https://github.com/openstack/cinder.git
openstack/glance:
git_ref: stable/mitaka
git_url: https://github.com/openstack/glance.git
openstack/heat:
git_ref: stable/mitaka
git_url: https://github.com/openstack/heat.git
openstack/horizon:
git_ref: stable/mitaka
git_url: https://github.com/openstack/horizon.git
openstack/keystone:
git_ref: stable/mitaka
git_url: https://github.com/openstack/keystone.git
openstack/neutron:
git_ref: stable/mitaka
git_url: https://github.com/openstack/neutron.git
openstack/nova:
git_ref: stable/mitaka
git_url: https://github.com/openstack/nova.git
openstack/requirements:
git_ref: stable/mitaka
git_url: https://git.openstack.org/openstack/requirements.git
openstack/sahara-dashboard:
git_ref: stable/mitaka
git_url: https://git.openstack.org/openstack/sahara-dashboard.git
openstack/murano-dashboard:
git_url: http://git.openstack.org/openstack/murano-dashboard.git
git_ref: stable/mitaka
openstack/searchlight-ui:
git_url: https://git.openstack.org/openstack/searchlight-ui.git
git_ref: stable/mitaka
openstack/designate-dashboard:
git_url: https://git.openstack.org/openstack/designate-dashboard.git
git_ref: stable/mitaka

View File

@ -0,0 +1,81 @@
images:
tag: mitaka
image_specs:
keystone:
tag: newton
horizon:
tag: newton
nova-upgrade:
tag: newton
nova-api:
tag: newton
nova-conductor:
tag: newton
nova-consoleauth:
tag: newton
nova-novncproxy:
tag: newton
nova-scheduler:
tag: newton
nova-compute:
tag: newton
nova-libvirt:
tag: newton
neutron-dhcp-agent:
tag: newton
neutron-l3-agent:
tag: newton
neutron-metadata-agent:
tag: newton
neutron-server:
tag: newton
neutron-openvswitch-agent:
tag: newton
glance-api:
tag: newton
glance-registry:
tag: newton
glance-upgrade:
tag: newton
sources:
openstack/cinder:
git_ref: stable/newton
git_url: https://github.com/openstack/cinder.git
openstack/glance:
git_ref: stable/newton
git_url: https://github.com/openstack/glance.git
openstack/heat:
git_ref: stable/newton
git_url: https://github.com/openstack/heat.git
openstack/horizon:
git_ref: stable/newton
git_url: https://github.com/openstack/horizon.git
openstack/keystone:
git_ref: stable/newton
git_url: https://github.com/openstack/keystone.git
openstack/neutron:
git_ref: stable/newton
git_url: https://github.com/openstack/neutron.git
openstack/nova:
git_ref: stable/newton
git_url: https://github.com/openstack/nova.git
openstack/requirements:
git_ref: stable/newton
git_url: https://git.openstack.org/openstack/requirements.git
openstack/sahara-dashboard:
git_ref: stable/newton
git_url: https://git.openstack.org/openstack/sahara-dashboard.git
openstack/murano-dashboard:
git_url: http://git.openstack.org/openstack/murano-dashboard.git
git_ref: stable/newton
openstack/searchlight-ui:
git_url: https://git.openstack.org/openstack/searchlight-ui.git
git_ref: stable/newton
openstack/designate-dashboard:
git_url: https://git.openstack.org/openstack/designate-dashboard.git
git_ref: stable/newton

View File

@ -0,0 +1,141 @@
.. _neutron_fuel_ccp_upgrade_report:
==================================================================
Fuel Containerized Control Plane performance report during upgrade
==================================================================
:Abstract:
This document includes OpenStack Networking (aka Neutron) performance test
results during upgrade from Mitaka to Newton. All tests have been performed
regarding
:ref:`neutron_fuel_ccp_upgrade_test_plans`
Environment description
=======================
For Kubernetes pre-deployment `Kargo tool`_ was used. More information about
*fuel-ccp* and how it can be installed can be found in
`official documentation`_.
.. _Kargo tool: https://github.com/kubespray/kargo
.. _official documentation: http://fuel-ccp.readthedocs.io/en/latest/
Hardware configuration of each server
-------------------------------------
All servers have configuration describing in table below
.. table:: Description of servers hardware
+-------+----------------+-------------------------------+
|server |vendor,model |HP,DL380 Gen9 |
+-------+----------------+-------------------------------+
|CPU |vendor,model |Intel,E5-2680 v3 |
| +----------------+-------------------------------+
| |processor_count |2 |
| +----------------+-------------------------------+
| |core_count |12 |
| +----------------+-------------------------------+
| |frequency_MHz |2500 |
+-------+----------------+-------------------------------+
|RAM |vendor,model |HP,752369-081 |
| +----------------+-------------------------------+
| |amount_MB |262144 |
+-------+----------------+-------------------------------+
|NETWORK|interface_name |ens4f1 |
| +----------------+-------------------------------+
| |vendor,model |Intel,X710 Dual Port |
| +----------------+-------------------------------+
| |bandwidth |10G |
| +----------------+-------------------------------+
| |interface_name |p1p1 |
| +----------------+-------------------------------+
| |vendor,model |Intel,X710 Dual Port |
| +----------------+-------------------------------+
| |bandwidth |10G |
+-------+----------------+-------------------------------+
|STORAGE|dev_name |/dev/sda |
| +----------------+-------------------------------+
| |vendor,model | | raid10 - HP P840 |
| | | | 12 disks EH0600JEDHE |
| +----------------+-------------------------------+
| |SSD/HDD |HDD |
| +----------------+-------------------------------+
| |size | 3,6TB |
+-------+----------------+-------------------------------+
Network configuration of each server
------------------------------------
All servers have same network configuration:
.. image:: configs/Network_Scheme.png
:alt: Network Scheme of the environment
:width: 650px
Fuel CCP configuration
----------------------
CCP main configuration
:download:`ccp.yaml <configs/ccp.yaml>`
CCP Cluster configuration
:download:`configs.yaml <configs/configs.yaml>`
CCP Cluster topology
:download:`topology.yaml <configs/topology.yaml>`
CCP repositories configuration
:download:`repos.yaml <configs/repos.yaml>`
CCP versions configuration for Mitaka
:download:`versions-mitaka.yaml <configs/versions-mitaka.yaml>`
CCP versions configuration for upgrade from Mitaka to Newton
:download:`versions-newton.yaml <configs/versions-newton.yaml>`
Upgrade and tests procedure
---------------------------
* Install fuel-ccp
* Rename ``ccp.yaml`` to ``/root/.ccp.ymal``
* Copy all yaml files to ``/root/``
* Rename ``versions-mitaka.yaml`` to ``versions.yaml``
* Execute ``ccp deploy`` and wait ok status for all services
* Install shaker
* On the node 172.20.9.40 execute ``shaker --scenario openstack/full_l2 --os-insecure --report full-l2.html --server-endpoint 172.20.9.40:31998``
* Wait message ``INFO shaker.engine.quorum [-] Finished processing operation: <shaker.engine.quorum.JoinOperation...`` from shaker
* Rename ``versions-newton.yaml`` to ``versions.yaml``
* Execute ``ccp deploy`` and wait ok status for all services
* Wait tests results from shaker
Test results
============
Shaker provides statistics about maximum, minimum and mean values of different
connection measurements. For each test was found the maximum among all maximum
values, minimum among all minimum values and counts the mean value from all
mean values.
Test Case 1: Analysis of L2 metrics during Open Stack upgrade
-------------------------------------------------------------
.. table:: Full L2 test results
+-------+----------------+---------------------------------------------------------+
|Losts |Errors |Full report |
+-------+----------------+---------------------------------------------------------+
|10 |0 |:download:`L2 Full tests report <reports/full-l2.html>` |
+-------+----------------+---------------------------------------------------------+
Test Case 2: Analysis of L3 metrics during Open Stack upgrade
-------------------------------------------------------------
.. table:: Full L3 East West test results
+-------+----------------+-----------------------------------------------------------------------------+
|Losts |Errors |Full report |
+-------+----------------+-----------------------------------------------------------------------------+
|0 |11 |:download:`L3 East West Full tests report <reports/full-l3-east-west.html>` |
+-------+----------------+-----------------------------------------------------------------------------+

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long