From 90eced9aa3a54122c85d164ce778662dfcfa15f3 Mon Sep 17 00:00:00 2001 From: Danil Golov Date: Wed, 5 Sep 2018 13:53:00 +0300 Subject: [PATCH] Introduce test case document for SRIOV functionality This commit contains test case document with steps that necessary to be done for checking proper work of SRIOV functionality in kuryr-kubernetes. Implements: blueprint kuryr-kubernetes-sriov-support Change-Id: I06f8b2c4d956f09cbf62aacebe68655c9c189b90 Signed-off-by: Danil Golov --- doc/source/installation/index.rst | 1 + doc/source/installation/sriov.rst | 2 + .../installation/testing_sriov_functional.rst | 244 ++++++++++++++++++ 3 files changed, 247 insertions(+) create mode 100644 doc/source/installation/testing_sriov_functional.rst diff --git a/doc/source/installation/index.rst b/doc/source/installation/index.rst index f94ddc5f4..5bcffd86f 100644 --- a/doc/source/installation/index.rst +++ b/doc/source/installation/index.rst @@ -46,3 +46,4 @@ This section describes how you can install and configure kuryr-kubernetes multi_vif_with_npwg_spec sriov testing_udp_services + testing_sriov_functional diff --git a/doc/source/installation/sriov.rst b/doc/source/installation/sriov.rst index e4d79ae38..8e1b6c688 100644 --- a/doc/source/installation/sriov.rst +++ b/doc/source/installation/sriov.rst @@ -1,3 +1,5 @@ +.. _sriov: + How to configure SR-IOV ports ============================= diff --git a/doc/source/installation/testing_sriov_functional.rst b/doc/source/installation/testing_sriov_functional.rst new file mode 100644 index 000000000..a22c8fac6 --- /dev/null +++ b/doc/source/installation/testing_sriov_functional.rst @@ -0,0 +1,244 @@ +Testing SRIOV functionality +=========================== + +Following the steps explained on :ref:`sriov` make sure that you have +already created and applied a ``NetworkAttachmentDefinition`` +containing a ``sriov`` driverType. Also make sure that +`sriov-device-plugin `_ +is enabled on the nodes. + +``NetworkAttachmentDefinition`` containing a ``sriov`` driverType might +look like: + +.. code-block:: yaml + + apiVersion: "k8s.cni.cncf.io/v1" + kind: NetworkAttachmentDefinition + metadata: + name: "net-sriov" + annotations: + openstack.org/kuryr-config: '{ + "subnetId": "88d0b025-2710-4f02-a348-2829853b45da", + "driverType": "sriov" + }' + +Here ``88d0b025-2710-4f02-a348-2829853b45da`` is an id of precreated +subnet that is expected to be used for SR-IOV ports: + +.. code-block:: bash + + $ neutron subnet-show 88d0b025-2710-4f02-a348-2829853b45da + +-------------------+--------------------------------------------------+ + | Field | Value | + +-------------------+--------------------------------------------------+ + | allocation_pools | {"start": "192.168.2.2", "end": "192.168.2.254"} | + | cidr | 192.168.2.0/24 | + | created_at | 2018-11-21T10:57:34Z | + | description | | + | dns_nameservers | | + | enable_dhcp | True | + | gateway_ip | 192.168.2.1 | + | host_routes | | + | id | 88d0b025-2710-4f02-a348-2829853b45da | + | ip_version | 4 | + | ipv6_address_mode | | + | ipv6_ra_mode | | + | name | sriov_subnet | + | network_id | 2f8b9103-e9ec-47fa-9617-0fb9deacfc00 | + | project_id | 92a4d7734b17486ba24e635bc7fad595 | + | revision_number | 2 | + | service_types | | + | subnetpool_id | | + | tags | | + | tenant_id | 92a4d7734b17486ba24e635bc7fad595 | + | updated_at | 2018-11-21T10:57:34Z | + +-------------------+--------------------------------------------------+ + +1. Create deployment definition with one +SR-IOV interface (apart from default one). Deployment definition +file might look like: + +.. code-block:: yaml + + apiVersion: extensions/v1beta1 + kind: Deployment + metadata: + name: nginx-sriov + spec: + replicas: 1 + template: + metadata: + name: nginx-sriov + labels: + app: nginx-sriov + annotations: + k8s.v1.cni.cncf.io/networks: net-sriov + spec: + containers: + - name: nginx-sriov + image: nginx + resources: + requests: + intel.com/sriov: '1' + cpu: "1" + memory: "512Mi" + limits: + intel.com/sriov: '1' + cpu: "1" + memory: "512Mi" + +Here ``net-sriov`` is the name of ``NetworkAttachmentDefinition`` +created before. + +2. Create deployment with the following command: + +.. code-block:: bash + + $ kubectl create -f + +3. Wait for the pod to get to Running phase. + +.. code-block:: bash + + $ kubectl get pods + NAME READY STATUS RESTARTS AGE + nginx-sriov-558db554d7-rvpxs 1/1 Running 0 1m + +4. If your image contains ``iputils`` (for example, busybox image), you can +attach to the pod and check that the correct interface has been attached +to the Pod. + +.. code-block:: bash + + $ kubectl get pod + $ kubectl exec -it nginx-sriov-558db554d7-rvpxs -- /bin/bash + $ ip a + +You should see default and eth1 interfaces. eth1 is the SR-IOV VF interface. + +.. code-block:: bash + + 1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever + inet6 ::1/128 scope host + valid_lft forever preferred_lft forever + 3: eth0@if43: mtu 1500 qdisc noqueue state UP qlen 1000 + link/ether fa:16:3e:1a:c0:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 192.168.0.9/24 scope global eth0 + valid_lft forever preferred_lft forever + inet6 fe80::f816:3eff:fe1a:c043/64 scope link + valid_lft forever preferred_lft forever + 13: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000 + link/ether fa:16:3e:b3:2e:70 brd ff:ff:ff:ff:ff:ff + inet 192.168.2.6/24 scope global eth1 + valid_lft forever preferred_lft forever + inet6 fe80::f816:3eff:fea8:55af/64 scope link + valid_lft forever preferred_lft forever + +4.1. Alternatively you can login to k8s worker and do the same from the +host system. +Use the following command to find out ID of running SR-IOV container: + +.. code-block:: bash + + $ docker ps + +Suppose that ID of created container is ``eb4e10f38763``. +Use the following command to get PID of that container: + +.. code-block:: bash + + $ docker inspect --format {{.State.Pid}} eb4e10f38763 + +Suppose that output of previous command is bellow: + +.. code-block:: bash + + $ 32609 + +Use the following command to get interfaces of container: + +.. code-block:: bash + + $ nsenter -n -t 32609 ip a + +You should see default and eth1 interfaces. eth1 is the SR-IOV VF interface. + +.. code-block:: bash + + 1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever + inet6 ::1/128 scope host + valid_lft forever preferred_lft forever + 3: eth0@if43: mtu 1500 qdisc noqueue state UP qlen 1000 + link/ether fa:16:3e:1a:c0:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 192.168.0.9/24 scope global eth0 + valid_lft forever preferred_lft forever + inet6 fe80::f816:3eff:fe1a:c043/64 scope link + valid_lft forever preferred_lft forever + 13: eth1: mtu 1500 qdisc pfifo_fast state UP qlen 1000 + link/ether fa:16:3e:b3:2e:70 brd ff:ff:ff:ff:ff:ff + inet 192.168.2.6/24 scope global eth1 + valid_lft forever preferred_lft forever + inet6 fe80::f816:3eff:fea8:55af/64 scope link + valid_lft forever preferred_lft forever + +In our example sriov interface has address 192.168.2.6 + +5. Use neutron CLI to check the port with exact address has been created on neutron: + +.. code-block:: bash + + $ openstack port list | grep 192.168.2.6 + +Suppose that previous command returns a list with one openstack port that +has ID ``545ec21d-6bfc-4179-88c6-9dacaf435ea7``. You can see its information +with the following command: + +.. code-block:: bash + + $ openstack port show 545ec21d-6bfc-4179-88c6-9dacaf435ea7 + +-----------------------+----------------------------------------------------------------------------+ + | Field | Value | + +-----------------------+----------------------------------------------------------------------------+ + | admin_state_up | UP | + | allowed_address_pairs | | + | binding_host_id | novactl | + | binding_profile | | + | binding_vif_details | port_filter='True' | + | binding_vif_type | hw_veb | + | binding_vnic_type | direct | + | created_at | 2018-11-26T09:13:07Z | + | description | | + | device_id | 7ab02cf9-f15b-11e8-bdf4-525400152cf3 | + | device_owner | compute:kuryr:sriov | + | dns_assignment | None | + | dns_name | None | + | extra_dhcp_opts | | + | fixed_ips | ip_address='192.168.2.6', subnet_id='88d0b025-2710-4f02-a348-2829853b45da' | + | id | 545ec21d-6bfc-4179-88c6-9dacaf435ea7 | + | ip_address | None | + | mac_address | fa:16:3e:b3:2e:70 | + | name | default/nginx-sriov-558db554d7-rvpxs | + | network_id | 2f8b9103-e9ec-47fa-9617-0fb9deacfc00 | + | option_name | None | + | option_value | None | + | port_security_enabled | False | + | project_id | 92a4d7734b17486ba24e635bc7fad595 | + | qos_policy_id | None | + | revision_number | 5 | + | security_groups | 1e7bb965-2ad5-4a09-a5ac-41aa466af25b | + | status | DOWN | + | subnet_id | None | + | updated_at | 2018-11-26T09:13:07Z | + +-----------------------+----------------------------------------------------------------------------+ + +The port would have the name of the pod, ``compute::kuryr::sriov`` for device owner and 'direct' vnic_type. +Verify that IP and MAC addresses of the port match the ones on the container. +Currently the neutron-sriov-nic-agent does not properly detect SR-IOV ports assigned to containers. This +means that direct ports in neutron would always remain in *DOWN* state. This doesn't affect the feature +in any way other than cosmetically.