From f98b182cb68564b58c930c923a3d2f20c9e85090 Mon Sep 17 00:00:00 2001 From: Peng Liu Date: Fri, 8 Jun 2018 16:29:30 +0800 Subject: [PATCH] Network Custom Resource Definition De-facto Standard support The purpose of Kubernetes Network Custom Resource Definition De-facto Standard spec defined by Network Plumbing Working Group, is trying to standardize the multi-network effort around K8S. In that spec, It defines a PoD annotation and CRD mechanism for attaching pods to multiple networks, using various CNIs. This proposal is based on VIF-Handler And Vif Drivers Design. A new VIF driver is created, which can parse the PoD annotation and CRD defined by NPWG spec, and return the vif objects to Multi-vif driver. Partially Implements: blueprint kuryr-npwg-spec-support Change-Id: I4402b9b8f04536b31f14684559c2aad86bd54162 --- doc/source/index.rst | 1 + doc/source/specs/rocky/npwg_spec_support.rst | 190 +++++++++++++++++++ 2 files changed, 191 insertions(+) create mode 100644 doc/source/specs/rocky/npwg_spec_support.rst diff --git a/doc/source/index.rst b/doc/source/index.rst index cb0bc24a6..29a8ccc8a 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -34,6 +34,7 @@ Design Specs specs/pike/fuxi_kubernetes specs/pike/sriov specs/queens/network_policy + specs/rocky/npwg_spec_support Indices and tables ------------------ diff --git a/doc/source/specs/rocky/npwg_spec_support.rst b/doc/source/specs/rocky/npwg_spec_support.rst new file mode 100644 index 000000000..9d888a34f --- /dev/null +++ b/doc/source/specs/rocky/npwg_spec_support.rst @@ -0,0 +1,190 @@ +========================================================================== +Kubernetes Network Custom Resource Definition De-facto Standard v1 Support +========================================================================== + +https://blueprints.launchpad.net/kuryr-kubernetes/+spec/kuryr-npwg-spec-support + +This spec proposes an approach to support the mechanism defined in Kubernetes +Network Custom Resource Definition De-facto Standard Version 1 [#]_, which is +used to attach multiple VIFs to Pods. + +Problem Description +------------------- + +There is always a desire for Pods to be able to be attached to multiple +interfaces in NFV use cases. However CNI plugins which have implemented this +functionality are using different way of defining the additional interfaces in +Pods. There is no standard approach among those CNI plugins. + +Therefore, the Networking Plumbing Working Group [#]_ drafted a spec (the NPWG +spec) trying to standardize the the way of attaching Pods to multiple networks. + +Proposed Change +--------------- + +The NPWG spec defines a "Network" Custom Resource object which describes how to +attach a Pod to the logical or physical network referenced by the object. + +The proposed change is based on VIF-Handler And Vif Drivers Design [#]_. A new +VIF driver 'npwg_multiple_interfaces' will be created to parse the annotation +of Pods and Network CRDs. The new VIF driver will be invoked by the multi-vif +driver as another sub-drivers. It should return a list of VIF objects. The +'npwg_multiple_interfaces' should invoke other VIF driver to create the vif +objects if it is necessary. + +The VIFHandler then updates the Pod annotation of 'openstack.org/kuryr-vif' +with the VIF objects. So that the Kuryr CNI can read these VIFs, and attaches +each of them to Pods namespace. If any of the additional interfaces failed to +be attached to the Pod, or any error happens during attachment, the CNI shall +return with error. + +Option in config file might look like this: + +.. code-block:: ini + + [kubernetes] + + enabled_vif_drivers = npwg_multiple_interfaces + +To define additional network in Pods, NPWG spec defines format of annotation. +Here's how a Pod Spec with additional networks requests might look like: + +.. code-block:: yaml + + kind: Pod + metadata: + name: my-pod + namespace: my-namespace + annotations: + kubernetes.v1.cni.cncf.io/networks: net-a,net-b,other-ns/net-c + +Or in JSON format like: + +.. code-block:: yaml + + kind: Pod + metadata: + name: my-pod + namespace: my-namespace + annotations: + kubernetes.v1.cni.cncf.io/networks: | + [ + {"name":"net-a"}, + {"name":"net-b"}, + { + "name":"net-c", + "namespace":"other-ns" + } + ] + +Then the VIF driver can parse the network information defined in 'Network' +objects. In NPWG spec, the 'Network' object definition is very flexible. +Implementations that are not CNI delegating plugins can add annotations to the +Network object and use those to store non-CNI configuration. And it is up to +the implementation to define the content it requires. + +Here is how 'Network' CRD specified in the NPWG spec. + +.. code-block:: yaml + + apiVersion: apiextensions.k8s.io/v1beta1 + kind: CustomResourceDefinition + metadata: + name: networks.kubernetes.cni.cncf.io + spec: + group: kubernetes.cni.cncf.io + version: v1 + scope: Namespaced + names: + plural: networks + singular: network + kind: Network + shortNames: + - net + validation: + openAPIV3Schema: + properties: + spec: + properties: + config: + type: string + +For Kuryr-kubernetes, users should define the 'Network' object with a Neutron +subnet created previously like: + +.. code-block:: yaml + + apiVersion: "kubernetes.cni.cncf.io/v1" + kind: Network + metadata: + name: a-bridge-network + annotations: + openstack.org/kuryr-config: '{ + "subnetId": "id_of_neutron_subnet_created_previously" + }' + +With information read from Pod annotation kubernetes.v1.cni.cncf.io/networks +and 'Network' objects, the Neutron ports could either be created or retrieved. +Then the Pod annotation openstack.org/kuryr-vif will be updated accordingly. + +Here's how openstack.org/kuryr-vif annotation with additional networks might +look like: + +.. code-block:: yaml + + kind: Pod + metadata: + name: my-pod + namespace: my-namespace + annotations: + openstack.org/kuryr-vif: { + # default interface remains intact + "eth0": { + ... Neutron vif object from default subnet ... + } + # additional interfaces appended by driver 'npwg_multiple_interfaces' + "eth1": { + ... Neutron vif object ... + } + "eth2": { + ... Neutron vif object ... + } + } + +Alternatives +~~~~~~~~~~~~ + +Currently, Kuryr-Kubernetes has already designed a way of defining additional +VIF. This spec will not change that part. Users can choose using which +format they want by configuring 'enabled_vif_drivers'. + +Other end user impact +~~~~~~~~~~~~~~~~~~~~~ +Pods always attach the default Kubernetes network as how Kuryr-Kubernetes works +today, and all networks specified in the Pod annotation are sidecars. + +Assignee(s) +~~~~~~~~~~~ + +Primary assignee: +Peng Liu + +Work Items +~~~~~~~~~~ + +* Implement a new NPWG spec compatible VIF driver. +* Document the procedure of using this new VIF driver. + +Possible Further Work +~~~~~~~~~~~~~~~~~~~~~ + +* To keep on track of the subsequent releases of NPWG spec. +* To allow defining new neutron network/subnet in 'Network' objects, so that + kuryr can create them in Neutron first, then attach Pod to it. + +References +---------- + +.. [#] https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit?usp=sharing +.. [#] https://groups.google.com/forum/?_escaped_fragment_=topic/kubernetes-sig-network/ANAjTyqVosw +.. [#] https://docs.openstack.org/kuryr-kubernetes/latest/devref/vif_handler_drivers_design.html