diff --git a/doc/source/test_plans/fuel_ccp_upgrade/plan.rst b/doc/source/test_plans/fuel_ccp_upgrade/plan.rst index ac0b8c6..2b9ca48 100644 --- a/doc/source/test_plans/fuel_ccp_upgrade/plan.rst +++ b/doc/source/test_plans/fuel_ccp_upgrade/plan.rst @@ -17,13 +17,9 @@ Fuel Containerized Control Plane upgradability performance testing Test Plan ========= -This document is inspired by -:ref:`openstack_control_plane_performance_test_plan`, and aims to cover -baseline cloud operations and extend this test suite to verify containerized -deployment approach. As :ref:`openstack_control_plane_performance_test_plan` -this test plan covers basic cloud operation VMs creation, as well as -long-running test suites to verify cloud stability during update Open -Stack from Mitaka to Newton. +This test plan covers basic network performance with long-running test suites +to verify cloud network stability and performance during update Open Stack +from Mitaka to Newton. Test Environment ---------------- diff --git a/doc/source/test_plans/neutron_fuel_ccp_upgrade/plan.rst b/doc/source/test_plans/neutron_fuel_ccp_upgrade/plan.rst new file mode 100644 index 0000000..45b8276 --- /dev/null +++ b/doc/source/test_plans/neutron_fuel_ccp_upgrade/plan.rst @@ -0,0 +1,189 @@ +.. _neutron_fuel_ccp_upgrade_test_plans: + +============================================================= +OpenStack Neutron Testing During Open Stack Upgrade Test Plan +============================================================= + +:status: **ready** +:version: 1.0 + +:Abstract: + + This test plan aims to provide set of tests to identify OpenStack + performance against given containerized OpenStack cloud (installed + on the top of pre-deployed Kubernetes cluster) using simple minimalistic set + of Shaker tests during upgrade from Mitaka to Newton. + +:Conventions: + + - **Shaker** - Data plane performance testing tool + - **iperf** - Commonly-used network testing tool + + +Test Plan +========= + +The purpose of this section is to describe scenarios for testing Open Stack +networking during upgrade from Mitaka to Newton. + + +Test Environment +---------------- + +Preparation +^^^^^^^^^^^ + +This test plan is performed against existing OpenStack cloud installed on top +of pre-deployed Kubernetes cluster with `fuel-ccp`_ tool with pre-installed Rally +framework. + +.. _fuel-ccp: http://fuel-ccp.readthedocs.io/en/latest/ + +Environment description +^^^^^^^^^^^^^^^^^^^^^^^ +The environment description includes hardware specification of servers, +network parameters, operation system and OpenStack deployment characteristics. + +Hardware +~~~~~~~~ + +This section contains list of all types of hardware nodes (table below is +an example). + ++-----------+-------+----------------------------------------------------+ +| Parameter | Value | Comments | ++-----------+-------+----------------------------------------------------+ +| model | | e.g. Supermicro X9SRD-F | ++-----------+-------+----------------------------------------------------+ +| CPU | | e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz | ++-----------+-------+----------------------------------------------------+ +| role | | e.g. compute or network | ++-----------+-------+----------------------------------------------------+ + +Network +~~~~~~~ + +This section contains list of interfaces and network parameters. For +complicated cases this section may include topology diagram and switch +parameters (table below is an example). + ++------------------+-------+-------------------------+ +| Parameter | Value | Comments | ++------------------+-------+-------------------------+ +| network role | | e.g. provider or public | ++------------------+-------+-------------------------+ +| card model | | e.g. Intel | ++------------------+-------+-------------------------+ +| driver | | e.g. ixgbe | ++------------------+-------+-------------------------+ +| speed | | e.g. 10G or 1G | ++------------------+-------+-------------------------+ +| MTU | | e.g. 9000 | ++------------------+-------+-------------------------+ +| offloading modes | | e.g. default | ++------------------+-------+-------------------------+ + +Software +~~~~~~~~ + +This section describes installed software (table below is an example). + ++-----------------+-------+---------------------------+ +| Parameter | Value | Comments | ++-----------------+-------+---------------------------+ +| OS | | e.g. Ubuntu 14.04.3 | ++-----------------+-------+---------------------------+ +| OpenStack | | e.g. Mitaka | ++-----------------+-------+---------------------------+ +| Hypervisor | | e.g. KVM | ++-----------------+-------+---------------------------+ +| Neutron plugin | | e.g. ML2 + OVS | ++-----------------+-------+---------------------------+ +| L2 segmentation | | e.g. VLAN / VxLAN / GRE | ++-----------------+-------+---------------------------+ +| virtual routers | | e.g. HA / DVR | ++-----------------+-------+---------------------------+ + +Test tool +--------- + +Shaker wraps around popular system network testing tools like iperf, iperf3 +and netperf (with help of flent). Shaker is able to deploy OpenStack instances +and networks in different topologies. Shaker scenario specifies the deployment +and list of tests to execute. Additionally tests may be tuned dynamically +in command-line. + +Test Case 1: Analysis of L2 metrics during Open Stack upgrade +------------------------------------------------------------- + +Description +^^^^^^^^^^^ + +`Shaker `__ is +able to deploy OpenStack instances and networks in different topologies. + + +The following steps should be executed: + +1. Deploy Kubernetes cluster +2. Deploy Open Stack cluster +3. Run full_l2 shaker's scenario and wait successfully stack heats deployment +4. Start Open Stack upgrade + + + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. table:: Shaker metrics + +======== =============== ================= ====================================== +Priority Value Measurement Units Description +======== =============== ================= ====================================== +1 Errors times Number of errors during testing +1 Lost times Number of lost results during testing +1 Latency ms The network latency +1 TCP bandwidth Mbits/s TCP network bandwidth +2 UDP bandwidth packets per sec Number of UDP packets of 32 bytes size +2 TCP retransmits packets per sec Number of retransmitted TCP packets +======== =============== ================= ====================================== + +Test Case 2: Analysis of L3 metrics during Open Stack upgrade +------------------------------------------------------------- + +Description +^^^^^^^^^^^ + +`Shaker `__ is +able to deploy OpenStack instances and networks in different topologies. + + +The following steps should be executed: + +1. Deploy Kubernetes cluster +2. Deploy Open Stack cluster +3. Run full_l3_east_west shaker's scenario and wait successfully stack heats + deployment +4. Start Open Stack upgrade + + + +List of performance metrics +^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.. table:: Shaker metrics + +======== =============== ================= ====================================== +Priority Value Measurement Units Description +======== =============== ================= ====================================== +1 Errors times Number of errors during testing +1 Lost times Number of lost results during testing +1 Latency ms The network latency +1 TCP bandwidth Mbits/s TCP network bandwidth +2 UDP bandwidth packets per sec Number of UDP packets of 32 bytes size +2 TCP retransmits packets per sec Number of retransmitted TCP packets +======== =============== ================= ====================================== + +Reports +======= + +Test plan execution reports: + * :ref:`neutron_fuel_ccp_upgrade_report` diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/Network_Scheme.png b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/Network_Scheme.png new file mode 100644 index 0000000..5ee45b5 Binary files /dev/null and b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/Network_Scheme.png differ diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/ccp.yaml b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/ccp.yaml new file mode 100644 index 0000000..9502261 --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/ccp.yaml @@ -0,0 +1,17 @@ +builder: + push: true + no_cache: false +registry: + address: "172.20.9.32:5000/env-1" +repositories: + skip_empty: True +kubernetes: + server: http://127.0.0.1:8080 + image_pull_policy: Always +--- +!include +- versions.yaml +- topology.yaml +- configs.yaml +- repos.yaml + diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/configs.yaml b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/configs.yaml new file mode 100644 index 0000000..6c39769 --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/configs.yaml @@ -0,0 +1,64 @@ +configs: + private_interface: p1p1.602 + public_interface: p1p1.602 + ingress: + enabled: true + glance: + tls: + enabled: false + bootstrap: + enable: true + neutron: + tls: + enabled: false + physnets: + - name: "physnet1" + bridge_name: "br-external" + interface: "bond0.648" + flat: true + vlan_range: false + dpdk: false + bootstrap: + internal: + enable: true + external: + enable: true + net_name: ext-net + subnet_name: ext-subnet + physnet: physnet1 + network: 172.20.172.0/22 + gateway: 172.20.172.1 + nameserver: 8.8.8.8 + pool: + start: 172.20.173.0 + end: 172.20.175.254 + keystone: + debug: true + security: + tls: + create_certificates: false + etcd: + tls: + enabled: false + db: + tls: + enabled: false + rabbitmq: + tls: + enabled: false + keystone: + tls: + enabled: false + cinder: + tls: + enabled: false + heat: + tls: + enabled: false + nova: + tls: + enabled: false + horizon: + tls: + enabled: false + diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/repos.yaml b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/repos.yaml new file mode 100644 index 0000000..2d75acb --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/repos.yaml @@ -0,0 +1,39 @@ +repositories: + repos: + - git_url: https://git.openstack.org/openstack/fuel-ccp-ceph + name: fuel-ccp-ceph + - git_url: https://git.openstack.org/openstack/fuel-ccp-debian-base + name: fuel-ccp-debian-base + - git_url: https://git.openstack.org/openstack/fuel-ccp-entrypoint + name: fuel-ccp-entrypoint + - git_url: https://git.openstack.org/openstack/fuel-ccp-etcd + name: fuel-ccp-etcd + - git_url: https://git.openstack.org/openstack/fuel-ccp-galera + name: fuel-ccp-galera + - git_url: https://git.openstack.org/openstack/fuel-ccp-glance + name: fuel-ccp-glance + - git_url: https://git.openstack.org/openstack/fuel-ccp-heat + name: fuel-ccp-heat + - git_url: https://git.openstack.org/openstack/fuel-ccp-horizon + name: fuel-ccp-horizon + - git_url: https://git.openstack.org/openstack/fuel-ccp-ironic + name: fuel-ccp-ironic + - git_url: https://git.openstack.org/openstack/fuel-ccp-keystone + name: fuel-ccp-keystone + - git_url: https://git.openstack.org/openstack/fuel-ccp-mariadb + name: fuel-ccp-mariadb + - git_url: https://git.openstack.org/openstack/fuel-ccp-memcached + name: fuel-ccp-memcached + - git_url: https://git.openstack.org/openstack/fuel-ccp-murano + name: fuel-ccp-murano + - git_url: https://git.openstack.org/openstack/fuel-ccp-neutron + name: fuel-ccp-neutron + - git_url: https://git.openstack.org/openstack/fuel-ccp-nginx + name: fuel-ccp-nginx + - git_url: https://git.openstack.org/openstack/fuel-ccp-nova + name: fuel-ccp-nova + - git_url: https://git.openstack.org/openstack/fuel-ccp-openstack-base + name: fuel-ccp-openstack-base + - git_url: https://git.openstack.org/openstack/fuel-ccp-rabbitmq + name: fuel-ccp-rabbitmq + diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/topology.yaml b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/topology.yaml new file mode 100644 index 0000000..cffb3cf --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/topology.yaml @@ -0,0 +1,86 @@ +nodes: + # node[1-3]: ETCD + # node[4-5]: Kubernetes master + + node([6|7|8])$: # 6-8 + roles: + - controller + - openvswitch + - messaging + - db + - indentity + + node9$: + roles: + - etc + - backups + - indentity + + node[1-9][0-9]$: # 10-99 + roles: + - compute + - openvswitch + - indentity + +replicas: + database: 3 + etcd: 1 + rpc: 3 + notifications: 3 + glance-api: 1 + glance-registry: 1 + keystone: 3 + nova-api: 3 + nova-scheduler: 3 + nova-conductor: 3 + neutron-server: 3 + neutron-metadata-agent: 3 + horizon: 3 + heat-api: 3 + heat-engine: 3 + +roles: + db: + - database + messaging: + - rpc + - notifications + controller: + - etcd + - glance-api + - glance-registry + - heat-api-cfn + - heat-api + - heat-engine + - horizon + - memcached + - neutron-dhcp-agent + - neutron-l3-agent + - neutron-metadata-agent + - neutron-server + - nova-api + - nova-conductor + - nova-consoleauth + - nova-novncproxy + - nova-scheduler + compute: + - nova-compute + - nova-libvirt + openvswitch: + - neutron-openvswitch-agent + - openvswitch-db + - openvswitch-vswitchd + etc: + - etcd + indentity: + - keystone + backups: + - backup +services: + database: + service_def: galera + rpc: + service_def: rabbitmq + notifications: + service_def: rabbitmq + diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-mitaka.yaml b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-mitaka.yaml new file mode 100644 index 0000000..df40105 --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-mitaka.yaml @@ -0,0 +1,41 @@ +images: + tag: mitaka +sources: + openstack/cinder: + git_ref: stable/mitaka + git_url: https://github.com/openstack/cinder.git + openstack/glance: + git_ref: stable/mitaka + git_url: https://github.com/openstack/glance.git + openstack/heat: + git_ref: stable/mitaka + git_url: https://github.com/openstack/heat.git + openstack/horizon: + git_ref: stable/mitaka + git_url: https://github.com/openstack/horizon.git + openstack/keystone: + git_ref: stable/mitaka + git_url: https://github.com/openstack/keystone.git + openstack/neutron: + git_ref: stable/mitaka + git_url: https://github.com/openstack/neutron.git + openstack/nova: + git_ref: stable/mitaka + git_url: https://github.com/openstack/nova.git + openstack/requirements: + git_ref: stable/mitaka + git_url: https://git.openstack.org/openstack/requirements.git + openstack/sahara-dashboard: + git_ref: stable/mitaka + git_url: https://git.openstack.org/openstack/sahara-dashboard.git + openstack/murano-dashboard: + git_url: http://git.openstack.org/openstack/murano-dashboard.git + git_ref: stable/mitaka + openstack/searchlight-ui: + git_url: https://git.openstack.org/openstack/searchlight-ui.git + git_ref: stable/mitaka + openstack/designate-dashboard: + git_url: https://git.openstack.org/openstack/designate-dashboard.git + git_ref: stable/mitaka + + diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-newton.yaml b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-newton.yaml new file mode 100644 index 0000000..5160dc2 --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/configs/versions-newton.yaml @@ -0,0 +1,81 @@ +images: + tag: mitaka + image_specs: + keystone: + tag: newton + + horizon: + tag: newton + + nova-upgrade: + tag: newton + nova-api: + tag: newton + nova-conductor: + tag: newton + nova-consoleauth: + tag: newton + nova-novncproxy: + tag: newton + nova-scheduler: + tag: newton + nova-compute: + tag: newton + nova-libvirt: + tag: newton + + neutron-dhcp-agent: + tag: newton + neutron-l3-agent: + tag: newton + neutron-metadata-agent: + tag: newton + neutron-server: + tag: newton + neutron-openvswitch-agent: + tag: newton + + glance-api: + tag: newton + glance-registry: + tag: newton + glance-upgrade: + tag: newton +sources: + openstack/cinder: + git_ref: stable/newton + git_url: https://github.com/openstack/cinder.git + openstack/glance: + git_ref: stable/newton + git_url: https://github.com/openstack/glance.git + openstack/heat: + git_ref: stable/newton + git_url: https://github.com/openstack/heat.git + openstack/horizon: + git_ref: stable/newton + git_url: https://github.com/openstack/horizon.git + openstack/keystone: + git_ref: stable/newton + git_url: https://github.com/openstack/keystone.git + openstack/neutron: + git_ref: stable/newton + git_url: https://github.com/openstack/neutron.git + openstack/nova: + git_ref: stable/newton + git_url: https://github.com/openstack/nova.git + openstack/requirements: + git_ref: stable/newton + git_url: https://git.openstack.org/openstack/requirements.git + openstack/sahara-dashboard: + git_ref: stable/newton + git_url: https://git.openstack.org/openstack/sahara-dashboard.git + openstack/murano-dashboard: + git_url: http://git.openstack.org/openstack/murano-dashboard.git + git_ref: stable/newton + openstack/searchlight-ui: + git_url: https://git.openstack.org/openstack/searchlight-ui.git + git_ref: stable/newton + openstack/designate-dashboard: + git_url: https://git.openstack.org/openstack/designate-dashboard.git + git_ref: stable/newton + diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/index.rst b/doc/source/test_results/neutron_fuel_ccp_upgrade/index.rst new file mode 100644 index 0000000..42bfbf1 --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/index.rst @@ -0,0 +1,141 @@ +.. _neutron_fuel_ccp_upgrade_report: + +================================================================== +Fuel Containerized Control Plane performance report during upgrade +================================================================== + +:Abstract: + + This document includes OpenStack Networking (aka Neutron) performance test + results during upgrade from Mitaka to Newton. All tests have been performed + regarding + :ref:`neutron_fuel_ccp_upgrade_test_plans` + +Environment description +======================= + +For Kubernetes pre-deployment `Kargo tool`_ was used. More information about +*fuel-ccp* and how it can be installed can be found in +`official documentation`_. + +.. _Kargo tool: https://github.com/kubespray/kargo +.. _official documentation: http://fuel-ccp.readthedocs.io/en/latest/ + +Hardware configuration of each server +------------------------------------- + +All servers have configuration describing in table below + +.. table:: Description of servers hardware + + +-------+----------------+-------------------------------+ + |server |vendor,model |HP,DL380 Gen9 | + +-------+----------------+-------------------------------+ + |CPU |vendor,model |Intel,E5-2680 v3 | + | +----------------+-------------------------------+ + | |processor_count |2 | + | +----------------+-------------------------------+ + | |core_count |12 | + | +----------------+-------------------------------+ + | |frequency_MHz |2500 | + +-------+----------------+-------------------------------+ + |RAM |vendor,model |HP,752369-081 | + | +----------------+-------------------------------+ + | |amount_MB |262144 | + +-------+----------------+-------------------------------+ + |NETWORK|interface_name |ens4f1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + | +----------------+-------------------------------+ + | |interface_name |p1p1 | + | +----------------+-------------------------------+ + | |vendor,model |Intel,X710 Dual Port | + | +----------------+-------------------------------+ + | |bandwidth |10G | + +-------+----------------+-------------------------------+ + |STORAGE|dev_name |/dev/sda | + | +----------------+-------------------------------+ + | |vendor,model | | raid10 - HP P840 | + | | | | 12 disks EH0600JEDHE | + | +----------------+-------------------------------+ + | |SSD/HDD |HDD | + | +----------------+-------------------------------+ + | |size | 3,6TB | + +-------+----------------+-------------------------------+ + + +Network configuration of each server +------------------------------------ + +All servers have same network configuration: + +.. image:: configs/Network_Scheme.png + :alt: Network Scheme of the environment + :width: 650px + +Fuel CCP configuration +---------------------- +CCP main configuration +:download:`ccp.yaml ` + +CCP Cluster configuration +:download:`configs.yaml ` + +CCP Cluster topology +:download:`topology.yaml ` + +CCP repositories configuration +:download:`repos.yaml ` + +CCP versions configuration for Mitaka +:download:`versions-mitaka.yaml ` + +CCP versions configuration for upgrade from Mitaka to Newton +:download:`versions-newton.yaml ` + +Upgrade and tests procedure +--------------------------- +* Install fuel-ccp +* Rename ``ccp.yaml`` to ``/root/.ccp.ymal`` +* Copy all yaml files to ``/root/`` +* Rename ``versions-mitaka.yaml`` to ``versions.yaml`` +* Execute ``ccp deploy`` and wait ok status for all services +* Install shaker +* On the node 172.20.9.40 execute ``shaker --scenario openstack/full_l2 --os-insecure --report full-l2.html --server-endpoint 172.20.9.40:31998`` +* Wait message ``INFO shaker.engine.quorum [-] Finished processing operation: ` | + +-------+----------------+---------------------------------------------------------+ + + + +Test Case 2: Analysis of L3 metrics during Open Stack upgrade +------------------------------------------------------------- +.. table:: Full L3 East West test results + + +-------+----------------+-----------------------------------------------------------------------------+ + |Losts |Errors |Full report | + +-------+----------------+-----------------------------------------------------------------------------+ + |0 |11 |:download:`L3 East West Full tests report ` | + +-------+----------------+-----------------------------------------------------------------------------+ + diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/reports/full-l2.html b/doc/source/test_results/neutron_fuel_ccp_upgrade/reports/full-l2.html new file mode 100644 index 0000000..55209b0 --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/reports/full-l2.html @@ -0,0 +1,724 @@ + + + + + + Shaker | The distributed data-plane testing tool for OpenStack + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+

Shaker | The Report

+
+
+
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ +
+ +
+ + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/source/test_results/neutron_fuel_ccp_upgrade/reports/full-l3-east-west.html b/doc/source/test_results/neutron_fuel_ccp_upgrade/reports/full-l3-east-west.html new file mode 100644 index 0000000..2bea30a --- /dev/null +++ b/doc/source/test_results/neutron_fuel_ccp_upgrade/reports/full-l3-east-west.html @@ -0,0 +1,724 @@ + + + + + + Shaker | The distributed data-plane testing tool for OpenStack + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+

Shaker | The Report

+
+
+
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ + +
+
+ +
+ +
+ + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file