After some interface operations (in particular the
``IpLinkCommand.set_ns`` operation), the network interface is
temporarily not present in the destination namespace. This patch
retries the interface "ip link set|show" command in that case.
Related-Bug: #1961740
Change-Id: I5a57cfc71ad59f1fe9ea65e19b1a32314d798729
The privileged/agent/linux/ip_lib.py code was not always
catching "entry does not exist" type errors when deleting
entries, and most of the callers were not catching it either,
which could lead to random failures.
Add code in the IP route, rule and bridge fdb code to catch
these errors and not raise on them, other exceptions will
still be raised.
Also fixed delete_neigh_entry() to not raise when the
given namespace does not exist to make it like all the
other calls in the file.
Added or modified functional tests for above cases.
Change-Id: I083649ab1b9a9057ee276a7f3ba069eb667db870
Closes-bug: #2030804
If get_link_devices() is called with 'index' in kwargs, pass the
argument to ip.get_links() correctly
Closes-Bug: #1953139
Change-Id: I6ae2d8c2a27aef548dd186e495c8998bf4086a20
This patch also removes the pylint disablement message control
statements and imports NetNS and IPRoute from the new locations
in pyroute2.
Trivial-Fix
Change-Id: I298a7da767473c236ddf03c5702a2904d4870284
Added ``devlink.get_port`` method that provides information about
a devlink port [1]. It is used to retrieve information about a port
representor connected to a local OVS instance (aka: hardware offloaded
ports). This method reports the PF PCI address, the PF index, the VF
index and the PF name; the PF name will be used to enforce the QoS
policies on the SR-IOV parent device (similar to what is done in the
ML2/SRIOV agent).
[1]https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html
Related-Bug: #1998608
Change-Id: I34daf554cabcf17cb6371d510d5827457012516d
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
some remaining ones in miscellaneous directories.
Also cleanup any remaining code that I missed in this
series, or has changed since I started.
Trivialfix
Change-Id: I17b4779020a7bfb369c3e721ab6638cd4a6ab50c
Created new add_ip_addresses privileged function
which takes an iterable of cidrs and adds them
in one privileged call. This is so we dont have to
take on additional priv overhead when calling
add_ip_address in a loop.
For parity, performed the same change on the
delete_ip_address function.
Closes-Bug: #1987281
Partial-Bug: #1981113
Change-Id: Ib1278af20c3b3b057712453cb249aba34b684a21
When a new IP route is created, before passing the route protocol,
find if it is a string and if this string is on the pyroute2 defined
protocols. In this case, pass the protocol number.
In the same way, when the IP route is returned, if the protocol is a
number, convert it to the corresponding protocol string.
Closes-Bug: #1988037
Change-Id: I4ca66d86705a55b2b63083c229629c16b6136283
"pyroute2" methods can include some objects that don't implement
any serialization method (e.g.: "nla_slot" [1]). In those methods
that require an output ("get_*", "list_*", etc.), the Neutron
IP library formats the output inside the privsep context only to
contain serializable objects.
However this library is also returning the blobs returned from
the "pyroute2" library, without parsing and formatting, from
methods that don't require an output ("set_*", "add_*", "delete_*",
etc.). This patch removes the "return" statement from those methods
because the output is not required and to avoid issues like those
reported in the related bug.
[1]8716b9b5c0/pyroute2/netlink/__init__.py (L1754)
Closes-Bug: #1986644
Change-Id: I491dbdabfda0ca010ca56355b71dfe150ed71a71
Fixed "ip route" commands to work with pyroute2 >= 0.6.10, that
introduces APIv2 for link, neighbour and route [1].
[1]bc0f5e2209
Closes-Bug: #1979031
Change-Id: Id2239b6827485a4d466b0916947428ceabef9139
pyroute 0.6.6 introduced a new exception NetlinkDumpInterrupted which
is raised when NLM_F_DUMP_INTR is set in the flags during dump of
devices.
The suggestion from pyroute developers is to retry in case of this
exception (see [1]).
[1]: https://github.com/svinota/pyroute2/issues/874#issuecomment-1063139555
Closes-Bug: #1962608
Change-Id: Ie195ad596fd148708fc30946bde964d52444afee
"ip link" commands allow to define VF rates independently.
That means, first "rate" (max BW) can be set and in a second
command "min" (min BW) (check LP bug description).
However Pyroute2 command to set the VF rates requires to set both.
If one value is missing ("min_tx_rate", "max_tx_rate"), the library
sets this value to 0; in other words, the value is deleted.
The Pyroute2 structures are built depending on the parameter names.
In this case, {'vf': {'rate': ...}} will create a "vf_rate" [1]
nla structure, that requires "min_tx_rate" and "max_tx_rate".
This is part of the full structure passed to the "iproute" library
[2].
This is an example of code that only sets the "max_tx_rate" for
the 15th VF of "enp196s0f0":
$ from neutron.plugins.ml2.drivers.mech_sriov.agent import pci_lib
pci = pci_lib.PciDeviceIPWrapper("enp196s0f0")
pci.set_vf_rate(15, {'max_tx_rate': 10})
The "msg" [3] (structure passed to "iproute" library) is this:
https://paste.opendev.org/show/b2FZBOebGOCHMrYhPr6X/. The
"min_tx_rate" is set to the default value 0.
This patch reads first the existing rates ("min_tx_rate",
"max_tx_rate") and populates the command parameters accordingly.
[1]a9564dff8e/pyroute2.core/pr2modules/netlink/rtnl/ifinfmsg/__init__.py (L712-L717)
[2]c8d9d92544/ip/ipaddress.c (L454-L470)
[3]a9564dff8e/pyroute2.core/pr2modules/iproute/linux.py (L1499)
Closes-Bug: #1962844
Change-Id: Ibbb6d938355440c42850812e368224b76b1fce19
This new context will have only two capabilities: CAP_NET_ADMIN
and CAP_SYS_ADMIN (for operations inside namespaces).
Change-Id: If9273db1a7ccdce3a81f68fce78408830e9c3d42
In order to dig the real action of a ResourceUpdate, add logs for:
1. add/update router
2. delete router
3. delete namespace
4. agent extension router add/delete/update actions
Change-Id: I5c0ff485cd0c966afe535f8063deca6e410e012d
Related-bug: #1881995
Since version 0.6.2, pyroute2 library dynamically imports the needed
modules when loaded. A static analysis will fail when checking the
import references.
Change-Id: I5aaf9494a2d5c2533199e6b92d4df8fe785f83a3
Closes-Bug: #1930750
"get_routing_table" uses "pyroute2.IPDB" that has been deprecated.
"list_ip_routes" has been improved to be able to read multipath
routes.
Closes-Bug: #1926476
Change-Id: I0299fa11a7afefbd2999f81cd4ed3beed572009c
This is a leftover of the "ip route" command migration to Pyroute2.
A new paremeter, "proto", is added to the IP route add and list
commands. The default protocol used is "static".
Story: #2007686
Task: #41284
Related-Bug: #1492714
Change-Id: I319fd0611d3e8a3a09d6d4e077a17a622f74f51c
Since [1], Pyroute forks the namespace creation to avoid calling
destructive routine "libc.unshare(CLONE_NEWNET)" from the main
process. This implementation uses sockets between both processes
to return any error feedback sent from the child process.
This patch implements the same fork without any communication. If
the child process raises an exception other than "OSError(EEXIST)",
the child process returns 1 that is read by the the main process,
that raises a "RuntimeError" exception.
Related-Bug: #1917487
[1]81db2c98a1
Change-Id: I0294586335a71d0757803843f675124bfb450967
It was marked that this try..except.. block can be removed when we will
be using pyroute2 >= 0.5.13. Now we have 0.5.13 in the lower-constraints
already so it's time to remove it.
Change-Id: Ic15361e34a2a9c371954d2f0851d230b8f9feb1f
As spotted in Focal testing patch [0], pep8 test fails with many
C0321 false-positives, reported in pylint as current version does not
support python 3.8 [1]
Use a newer version of pylint and astroid, fixing or disabling some of
the new checks: no-else-*, unnecessary-comprehension, import-outside-toplevel
[0] https://review.opendev.org/#/c/738163/
[1] https://github.com/PyCQA/pylint/issues/2737
Change-Id: Ie646b7093aa8634fd950c136a0eba9adcf56591c
Since [1], pyroute2 0.5.13 is supported.
In this new version, "link_lookup" do not raise a NetlinkError
exception if the device does not exist; instead returns an empty
list.
This patch handles both implementations.
[1]https://review.opendev.org/#/c/743277/
Change-Id: I77ef374ecb776966ea13499755777e2d763d884b
Closes-Bug: #1890353
Since [1], it's possible to specify the shared library to be used
when creating a Pyroute2 namespace context.
As commented in [2], "privsep" library makes use of eventlet to
implement multitasking. If the method executed returns the GIL,
nothing guarantees that the "eventlet" executor will return it
again to this task. This could lead to timeouts during the
execution of those methods.
From https://docs.python.org/3.6/library/ctypes.html#ctypes.PyDLL:
"Instances of this class behave like CDLL instances, except that
the Python GIL is not released during the function call, and
after the function execution the Python error flag is checked."
[1]https://github.com/svinota/pyroute2/issues/702
[2]https://review.opendev.org/#/c/717017/
Change-Id: I6c9f9adba8b4433cc96704bb69dd4e0d4b154ebd
Related-Bug: #1870352
That method could be used outside the library where currently is
implemented. This patch relocates it in a common place for all
privileged.agent.linux libraries.
Change-Id: I5a6124eca3b57ee36479c106b62d101f538c12eb
Story: #2007686
Task: #40047
Recent changes in some versions of iproute2 CLI output (v4.18),
have invalidated the regular expression used to parse the
"ip link" output.
To solve this problem and avoid future ones, pyroute2 is used to
retrieve the virtual functions information and set the VF attributes
(spoofcheck, min_tx_rate, max_tx_rate and link_state).
pyroute2 extended the "ip link" support to retrieve this information,
adding "ext_mask=1" in the get command. If no virtual functions are
present in this particular network interface, the added method,
"get_link_vfs", will return an empty list.
The set commands can return a "InterfaceOperationNotSupported" in
case the operation is not supported. For min_tx_rate, if the driver
does not support to set a minimum bandwidth, an "InvalidArgument"
(from a pyroute2.NetlinkError(22)) exception will be raised.
Change-Id: I680da4f64bd114f1caecaaeedbf8a4b1915a0849
Closes-Bug: #1878042
Some linux.ip_lib functions make use of "ctype.CDLL" methods
(create_netns, remove_netns). Those methods are called inside a
"privsep" context; that means the function reference and the
arguments are passed to a privileged context that will execute
the method.
"privsep" library makes use of eventlet to implement multitasking.
If the method executed returns the GIL, nothing guarantees that
the "eventlet" executor will return it again to this task. This
could lead to timeouts during the execution of those methods.
From https://docs.python.org/3.6/library/ctypes.html#ctypes.PyDLL:
"Instances of this class behave like CDLL instances, except that
the Python GIL is not released during the function call, and
after the function execution the Python error flag is checked."
Change-Id: I36ef9bf59e9c93f50464457a5d9a968738844079
Closes-Bug: #1870352
Ensure Pyroute2 netlink.nla_slot first element ("name"), is properly
converted to a string value. E.g.:
In: (b'IFA_ADDRESS', '192.168.30.20')
Out: ('IFA_ADDRESS', '192.168.30.20')
Python2 compatibility checks are removed from ip_lib.make_serializable.
Change-Id: I87efe6cf8734bae6523106944e99fbd7db9ef4d5
Related-Bug: #1846360
Because of issue with pyroute2.NetNS objects running in
threads we needed to lock privileged ip_lib functions
which are using this object to workaround the problem.
For details please check [1].
This problem was solved in pyroute 0.5.5. Now as we are using
0.5.7 we don't need those locks anymore.
[1] https://bugs.launchpad.net/neutron/+bug/1811515
Change-Id: I37da4025b93c8032164b0c3f12f400ed0d77c1ab
Load the glibc library only once, needed in the Pyroute2 methods to
create and delete a network namespace.
Change-Id: I95b7b7008f4788a98ef871c4b7aecea839ff2310
Closes-Bug: #1854462
In "ip_lib.ensure_device_is_ready", before retrieving the interface
attributes, a check is done to know if the interface exists. In case
it does not exist, the exception "NetworkInterfaceNotFound" will not
be raised and written in the logs.
Change-Id: I4b9fd0885d850601717274a5058e042871211bbb
Closes-Bug: #1854723
During the namespace PID listing, if a process is stopped, it will
disappear from "/proc/<pid>". When "os.stat(path)" is executed on
an unexistent path, returns "FileNotFoundError". This exception
should be catched and ignored.
Change-Id: Icde5e15bd97578f5ec8273f22ef8384502be1850
Related-Bug: #1841753
Instead of using exceptions as control flow, check the Pyroute2 command
result and only raise the Neutron exception if needed. This will also
reduce the traceback log in case of raising NetworkInterfaceNotFound.
Although in Python the use of exception for this is common, this is
usually considered an antipattern.
Change-Id: I0e8bb3b0f6a46f2bac75e38c6ac6cdd094247f89
Closes-Bug: #1849449
Pyroute2 can return information structures with values as bytes
instead of strings. In order to unify the output, those byte
parameters will be converted to string. This is needed in case of
having dictionaries with keys as bytes instead of strings.
Change-Id: Ic665b2c5e28d06e21190f06d9f3f8e8f8c3c792d
Closes-Bug: #1846360
Change the execution order of:
- @privileged.default.entrypoint
- @lockutils.synchronized("privileged-ip-lib")
"synchronized" decorator holds the execution of the function until
the lock is released. Using the current decorator ordering, this
active wait is done inside the privsep context. This can exhaust
the number of execution threads reserved for the privsep daemon.
Closes-Bug: #1833721
Change-Id: Ifcce954003e360f620f9131a36a08ab84cbe6193
In "NamespaceFixture", before deleting the namespace, this patch
introduces a check to first kill all processes running on it.
Closes-Bug: #1838793
Change-Id: I27f3db33f2e7ab685523fd2d6922177d7c9cb71b
- Add a new property to IPDevice to allow us to identify
the kind of the interface.
This change is required as an out of tree interface driver
which supports operations on a per-physnet basis
needs to be aware of the kind of interface an interface driver
created in order to correlate between an interface driver
and an interface created by it.
Change-Id: Icbdb011a639475f416ca1b98fdf3ce2f52482c7c
Partial-Bug: #1834176
Rodolfo Alonso Hernandez