It can happen that during the cleanup of KuryrPort when Pod is already
gone we'll fail trying to find the parent port ID. We have a bug that in
this case finalizing of KuryrPort fails.
This commit changes the way we look for the hostIP of the pod to
actually look up a node using the info from KuryrPort CRD. If this fails
(node removed?) we try querying OpenStack API to get this information.
If this fails too, we just don't pass hostIP to mocked Pod.
Change-Id: I72aea5713f90c8df2f5d0269fa83b8fdd5220c59
This got decided at the PTG. The code is old, not maintained, not tested
and most likely doesn't work anymore. Moreover it gave us a hard
dependency on grpcio and protobuf, which is fairly problematic in Python
and gave us all sorts of headaches.
Change-Id: I0c8c91cdd3e1284e7a3c1e9fe04b4c0fbbde7e45
There are some of the mentions of KuryrNetPolicy around our code. In
this patch we are removing it (with one exception - the spec for
originally designed CRD for network policy handling), just to avoid
confusion with currently used KuryrNetworkPolicy.
Change-Id: Ie9bb46467a249e1c0ada3a9810c4fff59fd57757
CRD KuryrNet was already replaced by KuryrNetwork, although there are
some spots, where it is mentioned - mostly docs and log messages. In
this commit we get rid of it once and for all.
Change-Id: I20345a1f4d4288534d620f0bd2196fc77ee795e9
The implementation have some difference with the description of
blueprint. For more strict isolation, we only get project id from
namespace annotaion or configure option. The other resources's
project id inherit it's project or get from configiure option.
Implements: blueprint specify-project-by-annotation
Change-Id: Ia82cce6b211226599b4e1ca0d10416ed5e519ea2
During removal of Neutron resources, sometimes there could be hanging
orphaned ports. Till now, all the removal was done one by one which
slows down removing process. In this change there is introduced removing
port in parallel in five concurrently run workers.
Change-Id: I74842989784601325b6d8977da4bc936ceedbc0e
It can happen that we get the CNI request, but pod gets deleted before
kuryr-controller was able to create KuryrPort for it. If kuryr-daemon
only watches for KuryrPort events it will not be able to notice that
and will wait until the timeout, which in effect doesn't play well with
some K8s tests.
This commit adds a separate Service that will watch on Pod events and if
Pod gets deleted we'll make sure to put a sentinel value (None) into the
registry so that the thread waiting for the KuryrPort to appear there
will know that it has to stop and raise an error.
Closes-Bug: 1963678
Change-Id: I52fc1805ec47f24c1da88fd13e79e928a3693419
Recent versions of cri-o and containerd are passing K8S_POD_UID as a CNI
argument, alongside with K8S_POD_NAMESPACE and K8S_POD_NAME. As both
latter variables cannot be used to safely identify a pod in the API
(StatefulSet recreates pods with the same name), we were prone to race
conditions in the CNI code that we could only workaround. The end effect
was mostly IP conflict.
Now that the UID argument is passed, we're able to compare the UID from
the request with the one in the API to make sure we're wiring the
correct pod. This commit implements that by making sure to move the
check to the code actually waiting for the pod to appear in the
registry. In case of K8S_POD_UID missing from the CNI request, API call
to retrieve Pod is used as a fallback.
We also know that this check doesn't work for static pods, so CRD and
controller needed to be updated to include information if the pod is
static on the KuryrPort spec, so that we can skip the check for the
static pods without the need to fetch Pod from the API.
Closes-Bug: 1963677
Change-Id: I5ef6a8212c535e90dee049a579c1483644d56db8
In case of hairpin LB traffic (member of the LB calls the LB and the
request is directed back to the same member) OVN replaces the source-ip
of the request with the LB IP. This means that pods with network
policies applied may have that traffic blocked when it should be
allowed.
To fix that this commit makes sure that SGs used for NPs include ingress
rules for each of the Service in it's namespace. It's not ideal but
seems to be a fair compromise between opening as little traffic as
possible and increasing number of security groups and rules.
As this commit makes sure all the NPs in the namespaces are reanalyzed
every time a Service is created or deleted, a little fixes in order to
support that are also made.
Change-Id: I7e0458c4071e4a43ab4d158429e05c67cd897a3c
Closes-Bug: 1923452
The timeout-client-data and timeout-member-data configurations
for Octavia listeners default to 50 seconds for load balancers
created by Kuryr. This patch allows the creation and modification
of load balancers handled by Kuryr with different timeouts values.
Implements: blueprint configure-lb-listeners-timeout
Change-Id: I99016001c2263023d1fa2637d7b5aeb23b3b2d9d
The Nampespace handler requires that the KuryrNetwork CRD
is created on the cluster instead of the old KuryrNet CRD.
This commit fixes the issue by checking if the KuryrNetwork
is enforced.
Closes-bug: 1916595
Change-Id: I1a5e19021c9dc7e8ddbb1898c23a4ed934497b2c
In order to support OpenShift's ability to run its nodes in various
OpenStack subnets in a dynamic way, this commit introduces the
OpenShiftNodesSubnets and MachineHandler. The idea is that
MachineHandler is responsible for watching the OpenShift Machine objects
and calling the driver. The driver will then save and serve a list of
current worker nodes subnets.
Change-Id: Iae3a5d011abaeab4aa97d6aa7153227c6f85b93c
Headless Service are not wired by Kuryr as it returns
directly the Pod's IPs behind the Service and do not
load-balance between them. This commit makes sure a klb
CR is not created for headless services as a LB won't
get created for it.
Change-Id: Ib389ddd5edca44c713149adc13486ab0b08007d2
This commit is a huge refactoring of how we handle network policies. In
general:
* KuryrNetPolicy is replaced by KuryrNetworkPolicy. The upgrade path
is handled in the constructor of KuryrNetworkPolicyHandler.
* New CRD has spec and status properties. spec is always populated by
NetworkPolicyHandler. status is handled by KuryrNetworkPolicyHandler.
This means that in order to trigger SG rules recalculation on Pod ang
Service events, the NetworkPolicy is "bumped" with a dummy annotation.
* NetworkPolicyHandler injects finalizers onto NetworkPolicy and
KuryrNetworkPolicy objects, so that objects cannot get removed before
KuryrNetworkPolicyHandler won't process deletion correctly.
Depends-On: https://review.opendev.org/742209
Change-Id: Iafc982e590ada0cd9d82e922c103583e4304e9ce
This commit adds support for creation of loadbalancer, listeners,
members, pools with using the CRD, it is also filling the status
field in the CRD.
Depends-On: https://review.opendev.org/#/c/743214/
Change-Id: I42f90c836397b0d71969642d6ba31bfb49786a43
Till now, we were using pod annotations to store information regarding
state of the associated VIFs to pod. This alone have its own issues and
it's prone to the inconsistency in case of controller failures.
In this patch we propose new CRD called KuryrPort for storage the
information about VIFs.
Depends-On: If639b63dcf660ed709623c8d5f788026619c895c
Change-Id: I1e76ea949120f819dcab6d07714522a576e426f2
This covers the scenario presented by k8s jobs in which the pods
are completed after the job and result in either Failed or Succeeded
state. Kubelet calls the CNI DEL for such pods so we should also delete
the neutron port and the corresponding annotation.
Closes-Bug: 1868074
Change-Id: I513784139b361da609f66dc2128970c8eae784c5
Signed-off-by: Nayan Deshmukh <n.deshmukh@samsung.com>
This patch moves the namespace handling to be more aligned
with the k8s style.
Depends-on: If0aaf748d13027b3d660aa0f74c4f6653e911250
Change-Id: Ia2811d743f6c4791321b05977118d0b4276787b5
When IPv6 and Network Policy are enabled we must ensure the
amphora SG is updated with sg rules using IPv6.
Implements: blueprint kuryr-ipv6-support
Change-Id: Id89b6c02e85d7faa75be6182c9d82ee7f32ff909
Previously to run DPDK applications on BM hosts with
SR-IOV functionality it was necessary to rebind virtual
function on some userspace driver (vfio-pci, uio_pci_generic
etc.).
But default Mellanox drivers (mlx4_core, mlx5_core) can work
with DPDK and do not require to be changed to userspace driver.
Moreover there could be occur a problem while trying to change
driver for Mellanox virtual functions.
This patch detects devices with Mellanox default driver in
the kuryr-kubernetes binding process and skip it (do not rebind).
Closes-Bug: 1856663
Change-Id: Ic5a76a6248a50370690dee2b0d57e2ded944a909
Signed-off-by: Danil Golov <d.golov@samsung.com>
Allowing user to configure the addtional_ifname_prefix allows
kuryr-kubernetes to work more smoothly with other CNI plugins (when
using multus or multiple chained CNIs for example).
Change-Id: I4ce35e939c1564c1ba0bd2ae55c09fa24a35c9ed
Signed-off-by: Yash Gupta <y.gupta@samsung.com>
This commit adds CRD to Kuryr that contains informations about
service's annotations and endpoints' annotations, also it is
ensured that CRD is created on the cluster when using devstack.
Partially-Implements: blueprint move-svc-annotations-to-crds
Change-Id: I4088ad09efe96b2d395fb892750adcb39abee3a4
This commit allows to use DPDK applications in
k8s pods.
To use DPDK applications inside the container
SR-IOV binding driver binds used device on apropriate
driver that is specified in config.
Partially implements: bp dpdk-applications-in-containers
Change-Id: I30ba1f30ae69562a91314fc2f14eba98b1692450
Signed-off-by: Danil Golov <d.golov@samsung.com>
This commit fixes incorrect way for choosing VF in
SR-IOV binding driver.
Previously sriov-device-plugin has choosen device
by it's own way while CNI choosed first available VF.
This entities did not know anything about each other's
choice.
Now SR-IOV binding driver gets a list of used by
kubelet devices with help of Pod Resources Client, then
chooses device from this list which is not used by pod
yet and passes an appropriate VF into container's
network namespace.
Also this commit contains tools for cluster upgrade.
Change-Id: I5b24981f715966369b05b8ab157f8bfe02afc2d4
Closes-Bug: 1826865
Signed-off-by: Danil Golov <d.golov@samsung.com>
When namespace subnet driver is used, a new subnet is created for
each new namespace. As pools are created per subnet, this patch
ensures that new ports are created for each pool for the new subnet
in the nested case.
Note this feature depends on using resource tagging to filter out
trunk ports in case of multiple clusters deployed on the same openstack
project or when other trunks are present. Otherwise it will consider
all the existing trunks no matter if they belong or not to the
kubernetes cluster.
NOTE: this is only for nested case, where pooling shows the greatest
improvements as ports are already ACTIVE.
Change-Id: Id014cf49da8d4cbe0c1795e47765fcf2f0684c09
This commit updates neutron sriov ports when
POD that contains it is in Running state.
It makes request to neutron API to update ports
with binding:profile information.
Due to this it is required to make actions with
privileged user having admin rights.
Change-Id: Iade446cefd78b494291729939db7db75692392c9
Closes-Bug: 1818606
Signed-off-by: Danil Golov <d.golov@samsung.com>
This commit allows to add pci information for direct
neutron ports attached to PODs into nodes annotations.
It happens on binding stage when pci information can
be requested.
Also this commit allows to delete annotations for such
ports when apropriate POD is deleted and VFs are returned
into host's network namespace.
For future commits: it is necessary to update neutron
ports with pci info when POD is in Running state.
Change-Id: I5ea8da6bb3143fd689e701f5e503488d4a6c9b33
Closes-Bug: 1818606
Signed-off-by: Danil Golov <d.golov@samsung.com>
This patch adds ability to work with new sriov device plugin's
feature. This feature implies arbitrary names for resource.
So kuryr-kubernetes should also know about such resources.
Resources are listed in /etc/pcidp/config.json
If physnet_resource_mappings is not specified it's assumed
resource name is physnet.
Implements: blueprint move-to-new-sriovdp
Change-Id: I07c5338b6d5b431435c467b0c52b50fe3385a86a
Signed-off-by: Alexey Perevalov <a.perevalov@samsung.com>
When a namespace is created, deleted or updated and
its labels matches the namespaceSelector of a NP,
the CRD and the respective sg must be updated.
Partially Implements: blueprint k8s-network-policies
Change-Id: I515de28647f5f06248555733c27dd4f5a56149ec
This commit adds support to match expressions and redesigns how
Security Groups are fetched and applied to pods that matches
a Network Policy.
Change-Id: I5d2831b09e24a2adc3c26aa29f1e7b131e9314ba
This patch adds a new handler in charge of reacting to pod relabeling
actions. It main purpose is to use it together with the network policy
handler and drivers to ensure the right policy is applied upon pod
label changes.
Partially Implements: blueprint k8s-network-policies
Change-Id: If026cefce847f77c54af09a0160eb35343f89f37
This patch adds health checks to Kuryr-Controller to
ensure that the CRDs that it depends on are defined
in the cluster.
Partially implements: blueprint k8s-network-policies
Change-Id: Ie3b6b6900d7a67402bddcca89b18dea35cc86992
This commit adds SR-IOV driver and new type of VIF to handle SR-IOV requests.
This driver can work as a primary driver and only one driver, but only when kubernetes
will fully support CNI specification.
Now this driver can work in couple with multi vif driver, e.g. NPWGMultiVIFDriver.
(see doc/source/installation/multi_vif_with_npwg_spec.rst)
Also this driver relies on kubernetes SRIOV device plugin.
This commit also adds 'default_physnet_subnets' setting, that should
include a mapping of physnets to neutron subnet IDs, it's necessary to
specify VIF's physnet (subnet id comes from annotation).
To get details how to create pods with sriov interfaces see
doc/source/installation/sriov.rst
Target bp: kuryr-kubernetes-sriov-support
Change-Id: I45c5f1a7fb423ee68731d0ae85f7171e33d0aeeb
Signed-off-by: Danil Golov <d.golov@partner.samsung.com>
Signed-off-by: Vladimir Kuramshin <v.kuramshin@samsung.com>
Signed-off-by: Alexey Perevalov <a.perevalov@samsung.com>
This commit implements NP driver actions for creating/updating SG and SG
rules. It also creates KuryrNetPolicy as a CRD so we don't have to rely
on the slow neutron API for time-costly operations such as listing SG
and so.
Security group rules and label matching will be handled in a follow-up
patch, as well as storing CRD object_id in a network policy annotation.
Unit tests will also be added after some more functionality is added
with the remaining patch series.
Partially-Implements: bp/k8s-network-policies
Change-Id: I6d45a462e812b24073b529144fc0843e8725a06e
This patch creates a npwg multi-vif driver which can parse the
Pod annotations and CRD defined in Network Plumbing Working
Group CRD SPEC.
Implements: blueprint kuryr-npwg-spec-support
Change-Id: I9ee9643b468a5fe453541b9cf1acf31ca872a313
This patch implements the multi-vif of VIF-Handler And Vif
Drivers Design.
This patch creates a new driver type MultiVIFDriver. It will
be the base class of real drivers like sriov,
additional_subnet and npwg_multiple_interfaces. Each of the
derived driver should implement the parsing of the additional
interfaces definition in K8S pods, and call VIF driver to
either create or acquire the Neutron port and its VIF object.
A list of enabled drivers can be returned by its class method.
So that the VIFHandler can invoke each driver one by one to
get the whole list of interfaces for one pod.
Partially Implements: blueprint multi-vif-pods
Change-Id: I8b5175a4637b18a0b574e27674a217865afb22b7
Signed-off-by: Peng Liu <pliu@redhat.com>
This patch adds a base driver and handler for network policy events. Follow up
patches will implement the driver and actions on network policies crud
actions, as well as tempest tests.
Partially Implements: blueprint k8s-network-policies
Co-Authored-By: Eyal Leshem <eyal.leshem@toganetworks.com>
Change-Id: I26969f2597c112259ca90724ff8b357bd8bb376e
This is the third patch of the Ingress Controller capability.
This patch implements OCP-Route and Ingress LBaaS handlers.
Those handlers should retrieve the L7 LB details from the
Ingress controller and update L7 policy/rules and pool/members
upon changes in OCP-route and k8S-endpoint resources.
Please follow the instructions below to verify
OCP-Router functionality:
https://docs.google.com/document/d/1c3mfBToBbWlwFcw3S8fr7pQZb5_YZqFYdlG1HqaQPkQ/edit?usp=sharing
Implements: blueprint openshift-router-support
Change-Id: Ibfb6cda6dde9613ad31859d38235be031ade0639
This is the second patch of the Ingress Controller capability.
In order for the K8S Ingress and OpenShift Route resources to work,
the cluster must have an Ingress Controller running.
This patch extends LBaaS driver to support L7 load balancing and
verifies, retrieves and stores the L7 router LB (pre-created by admin or
Devstack) details.
The OCP-route and K8S-endpoint handlers (implemented in next patch) will
query the ingress controller for the L7 router details.
Partially Implements: blueprint openshift-router-support
Change-Id: Id55169f6c9c1c607b2aa54c92711dfbd04a9e39d
This commit alters the format of the annotation, set and read by
controller/CNI. Instead of a single VIF object it now holds a
dictionary, that maps interface names to VIF objects. Controller
currently only sets 'eth0' as the default VIF. The CNI is altered
accordingly to read VIFs from the mapping and add all of them to Pods
namespace.
This commit does not include any mechanism to actually request multiple
IPs/ports for a Pod, but rather lays foundation for future commits, that
would allow that.
Related bp: kuryr-kubernetes-sriov-support
Targets bp: multi-vif-pods
Change-Id: Iaef928e7ab9dc0fce8b7e8fffb7b5a1f6b5ccb17
This patch adds a new subnet driver that creates a new network
for each created k8s namespace. It makes use of K8s CRDs to store
the information about the network resources created for each
namespace
Partially Implements: blueprint network-namespace
Change-Id: I7988e1da7a9ed57f29c85ddcd99bb2c87808010e
Since we started using Octavia we never got around to setting the
security groups for the legacy haproxy provider. This only affects when
using the native firewall as otherwise the haproxy internal ovs port
bypasses the SGs
Change-Id: Ie4a53dedf54472394f92fdfacddf0632e33f1f5b
Closes-Bug: 1749968
Co-Authored-By: Michał Dulko <mdulko@redhat.com>
Signed-off-by: Antoni Segura Puimedon <antonisp@celebdor.com>
This patch add the support to get the current status of the pools.
It can list the existing pools and their number of available
subports, as well as to show the ids of the ports associated to
a given pool.
Implements: blueprint kuryr-manager-cli-tool
Change-Id: I9332ffa259c4651b69c788985e5bbd1f98cb38cc
This patch introduces a ports pool manager that runs as an http server
thread belonging to the VIF handler. This Manager handles the requests
to populate and/or empty given subport pools so that they can be easily
managed.
It also includes a client and documentation to test/use the new functionality.
Implements: blueprint kuryr-manager-cli-tool
Change-Id: I495c0ca3ed997ab9da1763d8a3e60bbf7ac618b9
This patch introduces the modifications needed to support
L2 member mode communication between the loadbalancer and
the pool members (i.e., the pods belonging to the service).
It also includes the needed changes to set up the environment
with devstack.
Implements: blueprint octavia-layer2-member-connectivity
Change-Id: I345a71fbdf6f30f314b12aed1fc6f59177c03d00
Michał Dulko