8.3 KiB

Raw Permalink Blame History

Container SR-IOV networking

Problem description

SR-IOV (Single-root input/output virtualization) is a technique that allows a single physical PCIe (Peripheral Component Interconnect Express) device to be shared across several clients (VMs or containers). SR-IOV networking allows Nova VMs and containers access to virtual networks via SR-IOV NICs. Each such SR-IOV NIC would have a single PF (Physical Function) and multiple VFs (Virtual Functions). Essentially the PF and VFs appears as multiple PCIe SR-IOV based NICs. With SR-IOV networking, the traditional virtual bridge is no longer required, and thus higher networking performance can be achieved. This is an important requirement for most Virtualized Network Functions (VNFs). To support VNF application deployment over Zun containers, it is desirable to support SR-IOV networking feature for Zun.

To enable SR-IOV networking for Zun containers, Zun should provide a data model for PCI passthrough devices, and a filter to enable Zun scheduler locate the Zun computes that have access to the PCI passthrough devices. These two dependencies are addressed under separated blueprint[1][2].

Kuryr driver is used for Zun container networking. Kuryr implements a libnetwork remote network driver and maps its calls to OpenStack Neutron. It works as a translator between libnetwork's Container Network Model (CNM) and Neutron's networking model. Kuryr also acts as a libnetwork IPAM driver. This design will try to use the existing functions provided by Kuryr and identify the new requirements for Kuryr and Zun for the SR-IOV support.

With Kuryr driver, Zun will implement SR-IOV networking following the procedure below:

Cloud admin enables PCI-passthrough filter at /etc/zun.conf [1];
Cloud admin creates a new PCI-passthrough alias [2];
Cloud admin configures PCI passthrough whitelist at Zun compute [2];
Cloud admin enables sriovnicswitch on Neutron server (with existing Neutron support)[3];
Cloud admin configures supported_pci_vendor_devs at /etc/neutron/plugins/ml2/ml2_conf_sriov.ini (with existing Neutron support)[3];
Cloud admin configures sriov_nic.physical_device_mappings at /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on Zun compute nodes (with existing Neutron support)[3];
Cloud admin creates a network that SR-IOV NIC connects (with existing Neutron support)[3];
Cloud user creates a SR-IOV network port with an IP address (with existing Neutron support)[3]; Usually, the IP address is optional for creating a Neutron port. But Kuryr currently (in release Ocata) only support matching IP address to find the Neutron port that to be used for a container, thus IP address becomes a mandatory parameter here. This limitation will be removed once Kuryr can take neutron port-id as input.
Cloud user creates a new container bound with the SR-IOV network port.

This design spec focuses on the last step above.

Proposed change

Introduce a new option to allow user specify the neutron port-id when zun creates a container, for example:

zun create --name container_name --nets port=port_uuid ...

For a neutron SR-IOV port, the vnic_type attribute of the port is "direct". Ideally, kuryr_libnetwork should use the vnic_type to decide which port driver will be used to attach the neutron port to the container. However, kuryr can only support one port driver per host with current Ocata release. This means if we enable new SR-IOV port driver through the existing configuration at kuryr.conf, zun can only create containers using SR-IOV ports on the host. We expect kuryr will remove this limitation in the future.
Implement a new sriov_port_driver at kuryr_libnetwork.

The sriov_port_driver implements the abstract function create_host_iface(). The function allocates an SR-IOV VF on the host for the specified neutron port (e.g. pass-in parameter). Then the port is bound to the corresponding network subsystem[5].

The sriov_port_driver should also implement delete_host_iface(). The function de-allocates the SR-IOV VF and adds it back to the available VF pool.

The sriov_port_driver also implements get_container_iface_name(). This function should return the name of the VF instance.

Once a SR-IOV network port is available, Docker will call kuryr-libnetwork API to bind the neutron port to a network interface attached to the container. The name of the allocated VF instance will be passed to Docker[6]. The VF interface name representing the actual OS level VF interface that should be moved by LibNetwork into the sandbox[7].

Alternatives

Two other alternatives have been considered for the design. But both options requires significant changes in existing Kuryr implementations. The above design is believed to have minimum impact on Kuryr.

Option-1:

Zun creates containers with pre-configured SR-IOV port, and also manages the VF resources. This design option is very similar to the proposed design. The only difference is that VFs are managed or allocated by Zun. Zun then pass the VF name to Kuryr. The potential benefit of this option is to reuse all (PCI-passthrough and SR-IOV) resource management functions to be implemented for non-networking application.

Option-2:

Zun creates containers with both network and SR-IOV networking option. Kuryr driver will integrate with Docker SR-IOV driver[8] and create Docker network with SR-IOV VF interfaces as needed. This design offloads the SR-IOV specific implementation to the Docker SR-IOV driver, but at same time introduce additional work to integrate with the driver. With this design, the VF resources are managed by the Docker SR-IOV driver.

Data model impact

Refer to [2].

REST API impact

The proposed design relies an API change in container creation. The change allows user to specify an pre-created neutron port to be used for the container. The implementation of the change is in progress[9]. The existing container CRUD APIs will allow a set of new parameters for neutron networks with port-ID, for example:

"nets": [
    {
        "v4-fixed-ip": "",
        "network": "",
        "v6-fixed-ip": "",
        "port": "890699a9-4690-4bd6-8b70-3a9c1be77ecb"
    }
]