Add Support for DPDK Bonding
This patch will allow you to setup DPDK bonds within a given OVS bridge using DPDK-accelerated interfaces. A new provider network key, network_bond_interfaces, and related keys, have been introduced. Co-Authored-By: James Denton <james.denton@rackspace.com> Change-Id: I3fc2846a0c2b6579e4cdb54c3a7c36620700cd44
This commit is contained in:
Satish Patel
Dmitriy Rabotyagov
parent
782771f197
commit
0122ca1d65
|
|
@ -27,7 +27,7 @@ We recommend that you read the following documents before proceeding:
|
|||
* Open vSwitch with DPDK datapath:
|
||||
`<https://docs.openstack.org/neutron/latest/admin/config-ovs-dpdk.html>`_
|
||||
* Getting the best performance from DPDK:
|
||||
`<https://doc.dpdk.org/guides-16.04/linux_gsg/nic_perf_intel_platform.html>`_
|
||||
`<https://doc.dpdk.org/guides/linux_gsg/nic_perf_intel_platform.html>`_
|
||||
* OpenStack documentation on hugepages:
|
||||
`<https://docs.openstack.org/nova/latest/admin/huge-pages.html>`_
|
||||
|
||||
|
|
@ -156,11 +156,13 @@ The remaining cores can be reserved for virtual machine instances.
|
|||
|
||||
In this example, the breakdown would resemble the following:
|
||||
|
||||
```
|
||||
| Reserved Cores | Purpose | node0 | node1 |
|
||||
| ---------------------- | --------------------- | --------- | ----- |
|
||||
| 0,8,16,24 | Host Operating System | 0,16 | 8,24 |
|
||||
| 1,9,17,25 | DPDK PMDs | 1,17 | 9,25 |
|
||||
| 2-7,18-23 | Virtual Machines | 2-7,18-23 | N/A |
|
||||
```
|
||||
|
||||
The variables are overrides used to define this configuration are discussed
|
||||
in the following sections.
|
||||
|
|
@ -280,9 +282,11 @@ to a hexidecimal mask and defined using the ``ovs_dpdk_lcore_mask`` override.
|
|||
To convert to a hex mask you must first establish the binary mask of chosen
|
||||
cores using the following table:
|
||||
|
||||
```
|
||||
| 31 | 30 | . | 24 | 23 | . | 17 | 16 | 15 | . | 9 | 8 | 7 | . | 1 | 0 |
|
||||
| -- | -- | - | -- | -- | - | -- | -- | -- | - | -- | -- | -- | - | -- | -- |
|
||||
| 0 | 0 | . | 1 | 0 | . | 0 | 1 | 0 | . | 0 | 1 | 0 | . | 0 | 1 |
|
||||
```
|
||||
|
||||
The ellipses represent cores not shown. The binary mask for cores 0,8,16,24
|
||||
can be determined in the following way:
|
||||
|
|
@ -302,9 +306,11 @@ file or ``openstack_user_config.yml``:
|
|||
The mask for cores 1,9,17,25 reserved for DPDK PMDs can be determined in
|
||||
a similar fashion. The table would resemble the following:
|
||||
|
||||
```
|
||||
| 31 | 30 | . | 25 | 24 | . | 17 | 16 | 15 | . | 9 | 8 | 7 | . | 1 | 0 |
|
||||
| -- | -- | - | -- | -- | - | -- | -- | -- | - | -- | -- | -- | - | -- | -- |
|
||||
| 0 | 0 | . | 1 | 0 | . | 1 | 0 | 0 | . | 1 | 0 | 0 | . | 1 | 0 |
|
||||
```
|
||||
|
||||
The ellipses represent cores not shown. The binary mask for cores 1,9,17,254
|
||||
can be determined in the following way:
|
||||
|
|
@ -401,6 +407,12 @@ Specify provider network definitions in your
|
|||
``/etc/openstack_deploy/openstack_user_config.yml`` that define one or more
|
||||
Neutron provider bridges and related configuration:
|
||||
|
||||
.. note::
|
||||
|
||||
Bridges specified here will be created automatically. If *network_interface*
|
||||
is defined, the interface will be placed into the bridge automatically
|
||||
as a *DPDK-accelerated* interface.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- network:
|
||||
|
|
@ -409,13 +421,55 @@ Neutron provider bridges and related configuration:
|
|||
type: "vlan"
|
||||
range: "101:200,301:400"
|
||||
net_name: "physnet1"
|
||||
network_interface: "eno49"
|
||||
group_binds:
|
||||
- neutron_openvswitch_agent
|
||||
|
||||
.. note::
|
||||
A *DPDK-accelerated* **bond** interface can be created by specifying a list
|
||||
of member interfaces using `network_bond_interfaces`. The bond port will
|
||||
be created automatically and added to the respective bridge in OVS:
|
||||
|
||||
A single DPDK interface can be connected to an OVS provider bridge, and
|
||||
must be done using the ``ovs-vsctl`` command as a post-installation step.
|
||||
.. code-block:: yaml
|
||||
|
||||
- network:
|
||||
container_bridge: "br-provider"
|
||||
container_type: "veth"
|
||||
type: "vlan"
|
||||
range: "101:200,301:400"
|
||||
net_name: "physnet1"
|
||||
network_bond_interfaces:
|
||||
- "0000:04:00.0"
|
||||
- "0000:04:00.1"
|
||||
group_binds:
|
||||
- neutron_openvswitch_agent
|
||||
|
||||
Additional OVS bond parameters can be specified using the following keys:
|
||||
|
||||
* bond_mode (Default: active-backup)
|
||||
* lacp (Default: off)
|
||||
* bond_downdelay (Default: 100)
|
||||
* bond_updelay (Default: 100)
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
- network:
|
||||
container_bridge: "br-provider"
|
||||
container_type: "veth"
|
||||
type: "vlan"
|
||||
range: "101:200,301:400"
|
||||
net_name: "physnet1"
|
||||
network_bond_interfaces:
|
||||
- "0000:04:00.0"
|
||||
- "0000:04:00.1"
|
||||
bond_mode: balance-tcp
|
||||
lacp: active
|
||||
bond_downdelay: 200
|
||||
bond_updelay: 200
|
||||
group_binds:
|
||||
- neutron_openvswitch_agent
|
||||
|
||||
For more information on possible values, visit:
|
||||
`<https://docs.ansible.com/ansible/latest/collections/openvswitch/openvswitch/openvswitch_bond_module.html>`_
|
||||
|
||||
Set the following user variables in your
|
||||
``/etc/openstack_deploy/user_variables.yml`` to enable the Open vSwitch driver
|
||||
|
|
@ -430,7 +484,9 @@ and DPDK support:
|
|||
ovs_dpdk_support: True
|
||||
|
||||
# Add these overrides or set on per-host basis in openstack_user_config.yml
|
||||
ovs_dpdk_pci_addresses: "0000:03:00.0"
|
||||
ovs_dpdk_pci_addresses:
|
||||
- "0000:04:00.0"
|
||||
- "0000:04:00.1"
|
||||
ovs_dpdk_lcore_mask: 1010101
|
||||
ovs_dpdk_pmd_cpu_mask: 2020202
|
||||
ovs_dpdk_socket_mem: "1024,1024"
|
||||
|
|
@ -442,20 +498,29 @@ and DPDK support:
|
|||
Post-installation
|
||||
~~~~~~~~~~~~~~~~~
|
||||
|
||||
Once the playbooks have been run and OVS/DPDK has been configured, it will be
|
||||
Once the playbooks have been run and OVS/DPDK has been configured, it may be
|
||||
necessary to add a physical interface to the provider bridge before networking
|
||||
can be fully established.
|
||||
can be fully established *if* `network_interface` or `network_bond_interfaces`
|
||||
have not been defined.
|
||||
|
||||
On compute nodes, the following command can be used to attach a NIC port
|
||||
``0000:03:00.0`` to the provider bridge ``br-provider``:
|
||||
``0000:04:00.0`` to the provider bridge ``br-provider``:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ovs-vsctl add-port br-provider 0000:03:00.0 -- set Interface 0000:03:00.0 type=dpdk options:dpdk-devargs=0000:03:00.0
|
||||
ovs-vsctl add-port br-provider 0000:04:00.0 -- set interface 0000:04:00.0 type=dpdk options:dpdk-devargs=0000:04:00.0
|
||||
|
||||
The command can be adjusted according to your configuration.
|
||||
Additionally, it may be necessary to make post-installation adjustments to
|
||||
interface queues or other parameters to avoid errors within Open vSwitch:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
ovs-vsctl set interface 0000:04:00.0 options:n_txq=5
|
||||
ovs-vsctl set interface 0000:04:00.0 options:n_rxq=5
|
||||
|
||||
The command(s) can be adjusted according to your configuration.
|
||||
|
||||
.. warning::
|
||||
|
||||
Adding multiple ports to the bridge may result in bridging loops unless
|
||||
bonding is configured. DPDK bonding is outside the scope of this guide.
|
||||
Adding multiple ports to a bridge may result in bridging loops unless
|
||||
bonding is configured.
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ from ansible.module_utils.basic import AnsibleModule
|
|||
DOCUMENTATION = """
|
||||
---
|
||||
module: provider_networks
|
||||
version_added: "1.8.6"
|
||||
version_added: "1.8.7"
|
||||
short_description:
|
||||
- Parse a list of networks and return data that Ansible can use
|
||||
description:
|
||||
|
|
@ -116,6 +116,22 @@ EXAMPLES = """
|
|||
# group_binds:
|
||||
# - neutron_openvswitch_agent
|
||||
# - network:
|
||||
# container_bridge: "br-provider"
|
||||
# container_type: "veth"
|
||||
# container_interface: "eth11"
|
||||
# network_bond_interfaces:
|
||||
# - "0000:02:00.0"
|
||||
# - "0000:02:00.1"
|
||||
# bond_mode: balance-tcp
|
||||
# bond_updelay: 100
|
||||
# bond_downdelay: 100
|
||||
# lacp: active
|
||||
# type: "vlan"
|
||||
# range: "1:1, 101:101"
|
||||
# net_name: "physnet1"
|
||||
# group_binds:
|
||||
# - neutron_openvswitch_agent
|
||||
# - network:
|
||||
# container_bridge: "br-storage"
|
||||
# container_type: "veth"
|
||||
# container_interface: "eth2"
|
||||
|
|
@ -157,6 +173,14 @@ EXAMPLES = """
|
|||
# "flat:brx-eth12",
|
||||
# "vlan:brx-eth11"
|
||||
# ],
|
||||
# "network_bond_interfaces_mappings": [{
|
||||
# 'bridge': 'br-provider',
|
||||
# 'interfaces': ['0000:02:00.0', '0000:02:00.1'],
|
||||
# 'bond_mode': 'balance-tcp',
|
||||
# 'lacp': 'active',
|
||||
# 'bond_updelay': 100,
|
||||
# 'bond_downdelay': 100
|
||||
# }],
|
||||
# "network_sriov_mappings": "physnet1:p1p1,physnet1:p1p2",
|
||||
# "network_sriov_mappings_list": [
|
||||
# "physnet1:p1p1"
|
||||
|
|
@ -202,6 +226,7 @@ class ProviderNetworksParsing(object):
|
|||
self.network_types = list()
|
||||
self.network_sriov_mappings = list()
|
||||
self.network_interface_mappings = list()
|
||||
self.network_bond_interfaces_mappings = list()
|
||||
|
||||
def load_networks(self, provider_networks, is_metal=False,
|
||||
bind_prefix=None, group_names=None):
|
||||
|
|
@ -299,6 +324,29 @@ class ProviderNetworksParsing(object):
|
|||
)
|
||||
)
|
||||
|
||||
# Builds a list of provider bridge to physical
|
||||
# interface (bond member) mappings and is used
|
||||
# when constructing OVS bonds
|
||||
if 'network_bond_interfaces' in net['network']:
|
||||
self.network_bond_interfaces_mappings.append({
|
||||
'bridge': net['network'][
|
||||
'container_bridge'
|
||||
],
|
||||
'interfaces': net['network'][
|
||||
'network_bond_interfaces'
|
||||
],
|
||||
'bond_mode': net['network'].get(
|
||||
'bond_mode', 'active-backup'
|
||||
),
|
||||
'lacp': net['network'].get('lacp', 'off'),
|
||||
'bond_updelay': net['network'].get(
|
||||
'bond_updelay', 100
|
||||
),
|
||||
'bond_downdelay': net['network'].get(
|
||||
'bond_downdelay', 100
|
||||
)
|
||||
})
|
||||
|
||||
# SR-IOV interface mappings
|
||||
if 'sriov_host_interfaces' in net['network']:
|
||||
host_interfaces = \
|
||||
|
|
@ -367,7 +415,12 @@ def main():
|
|||
'network_interface_mappings': ','.join(
|
||||
pnp.network_interface_mappings
|
||||
),
|
||||
'network_interface_mappings_list': pnp.network_interface_mappings
|
||||
'network_interface_mappings_list': pnp.network_interface_mappings,
|
||||
'network_bond_interfaces_mappings': ','.join(
|
||||
map(str, pnp.network_bond_interfaces_mappings)
|
||||
),
|
||||
'network_bond_interfaces_mappings_list':
|
||||
pnp.network_bond_interfaces_mappings
|
||||
}
|
||||
|
||||
module.exit_json(changed=True, **resp)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,10 @@
|
|||
---
|
||||
features:
|
||||
- |
|
||||
The ``provider_networks`` library has been updated to support the
|
||||
definition of bond member interfaces that can automatically be added as
|
||||
bond ports to OVS provider bridges setup during a deployment. This
|
||||
feature is currently limited to DPDK-based deployments. To activate this
|
||||
feature, add the ``network_bond_interfaces`` key to the respective provider
|
||||
network definition in ``openstack_user_config.yml``. For more information,
|
||||
refer to the latest Open vSwitch w/ DPDK deployment guide.
|
||||
|
|
@ -28,6 +28,7 @@
|
|||
dest: "/etc/dpdk/interfaces"
|
||||
owner: "root"
|
||||
group: "root"
|
||||
mode: "0640"
|
||||
when:
|
||||
- neutron_services['neutron-openvswitch-agent']['group'] in group_names
|
||||
- '"nova_compute" in group_names'
|
||||
|
|
@ -38,6 +39,7 @@
|
|||
dest: "/etc/dpdk/dpdk.conf"
|
||||
owner: "root"
|
||||
group: "root"
|
||||
mode: "0640"
|
||||
when:
|
||||
- neutron_services['neutron-openvswitch-agent']['group'] in group_names
|
||||
- '"nova_compute" in group_names'
|
||||
|
|
@ -111,7 +113,7 @@
|
|||
- neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr']
|
||||
- neutron_provider_networks.network_mappings is defined
|
||||
|
||||
# (todo) Loop thru ints or build a bond with ints. TBD.
|
||||
# Adds a single host interface to an OVS bridge
|
||||
- name: Add ports to Network Provider Bridges
|
||||
openvswitch_port:
|
||||
bridge: "{{ interface_mapping.split(':')[0] }}"
|
||||
|
|
@ -125,3 +127,23 @@
|
|||
- neutron_services['neutron-openvswitch-agent']['group'] in group_names
|
||||
- neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr']
|
||||
- neutron_provider_networks.network_interface_mappings is defined and (neutron_provider_networks.network_interface_mappings|length > 0)
|
||||
|
||||
# Adds a DPDK-accelerated bond interface to an OVS bridge
|
||||
- name: Add Bonds to Network Provider Bridges
|
||||
openvswitch.openvswitch.openvswitch_bond:
|
||||
bridge: "{{ bond_interfaces_mapping.bridge }}"
|
||||
port: "{{ bond_interfaces_mapping.bridge }}-dpdkbond"
|
||||
interfaces: "{{ bond_interfaces_mapping.interfaces }}"
|
||||
bond_mode: "{{ bond_interfaces_mapping.bond_mode | default('active-backup') }}"
|
||||
lacp: "{{ bond_interfaces_mapping.lacp | default('off') }}"
|
||||
bond_updelay: "{{ bond_interfaces_mapping.bond_updelay | default(100) }}"
|
||||
bond_downdelay: "{{ bond_interfaces_mapping.bond_downdelay | default(100) }}"
|
||||
set: "{% for interface in bond_interfaces_mapping.interfaces %}interface {{ interface }} type=dpdk options:dpdk-devargs='{{ interface }}'{% if not loop.last %},{% endif %}{% endfor %}"
|
||||
state: present
|
||||
with_items: "{{ neutron_provider_networks.network_bond_interfaces_mappings }}"
|
||||
loop_control:
|
||||
loop_var: bond_interfaces_mapping
|
||||
when:
|
||||
- neutron_services['neutron-openvswitch-agent']['group'] in group_names
|
||||
- neutron_plugin_type in ['ml2.ovs', 'ml2.ovs.dvr']
|
||||
- neutron_provider_networks.network_bond_interfaces_mappings is defined and (neutron_provider_networks.network_bond_interfaces_mappings|length > 0)
|
||||
|
|
|
|||
Loading…
Reference in New Issue