The general approach is to update the ovn_client and ovn_db_sync
code to handle multiple gateway ports wherever a single gateway
port is handled today.
In this patch set multiple static routes for default gateway will
be added by default when multiple gw ports are present. Support
for the `enable_default_route_ecmp` attribute to control this
behavior will be added in subsequent patch set to avoid making
this change too large.
Partial-Bug: #2002687
Change-Id: I00b1f29172be5a0034b921b11af3a8d502273766
This patch removes the compatibility with OVN under v20.09. That
implies the OVN Southbound definition has "Chassis_Private" table.
Any previous check is removed from the code.
This patch also adds a sanity check, testing that the OVN Southbound
database definition is greater or equal to 2.9.0 [1].
The testing OVN NB and SB schemas are updated to the files contained in
OVN v22.09. The new testing NB schema version is 6.3.9; the new testing
SB schema version is 20.25.0.
[1]4adc10f581
Closes-Bug: #2002839
Change-Id: Iec8854749a1df81eb6a7154d3f951e176c69156d
This patch introduces a new configuration for OVN CMS Options called
"enable-chassis-as-extport-host". This configuration can be used
by ML2/OVN to identify nodes that are eligible for scheduling OVN's
external ports.
Prior to this patch, external ports were always scheduled on centralized
networked nodes tagged with the "enable-chassis-as-gw" flag in the OVN
CMS Options but, when it comes to deploying OpenStack on OpenShift
requiring services such as the OVN Metadata Agent or DHCP Agent to
serve those external ports and running them on control plane nodes are
not ideal. This is where this patch comes handy allowing these ports to
have more flexibility in where they are scheduled.
The patch is also backward compatible and if the new configuration is
not present on the OVN CMS Options, ML2/OVN will continue to schedule
the external ports on nodes configured with the previous configuration
like always.
Documentation will be updated on a follow up patch.
Closes-Bug: 2037294
Change-Id: Ic46d847e3aebfe543d5a7ab49d18d1f1abf1342e
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
Default SG rules created as template in the Neutron DB are now used to
create security group rules for each new default and non-default SG
created in Neutron.
Closes-bug: #1983053
Change-Id: Iaf27deb955c3844409fcd36239511478e9607a82
This patch implements a more resilient approach to handle the case
where Neutron API workers are killed and restarted. Instead of marking
all nodes for that host as offline, this patch tries to remove the
worker that was killed from the Hash Ring leaving all others nodes for
that host online.
In case the we fail to remove the node and another entry is added upon the
restart of the worker this patch also logs a clear critical log message to
alert the operator that there are more Hash Ring nodes than API workers
(it's expect to be the same) and that OVSDB events could go missing if
they are routed to the previous node that failed to be removed from the
ring.
Closes-Bug: #2024205
Change-Id: I4b7376cf7df45fcc6e487970b068d06b4e74e319
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
This patch implements the proposed solution from LP #2024205 where upon
a Neutron being killed, it could trigger the deletion of the entries
from the ovn_hash_ring table that matches the server hostname. When
this happens on all controllers this could lead to the ovn_hash_ring
being rendered empty which will result in ML2/OVN not processing any
OVSDB events.
Instead of removing the nodes from the ovn_hash_ring table at exit, this
patch changes the code to just mark them as offline instead. That way,
the nodes will remain registered in the table and the heartbeat thread
will set them as online again on the next beat. If the service is
stopped properly there won't be any heartbeat anymore and the nodes will
be seeing as offline by the Hash Ring Manager (same as if they were
deleted).
For more info see LP #2024205.
Closes-Bug: #2024205
Change-Id: I052841c87651773c4988fcf39f9f978094297704
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
This reverts commit be4e150de9.
Reason for revert: some CI gates (devstack for example) are still
using some OS (Ubuntu Focal) that provide older OVN version. On
these jobs we are not manually compiling OVN.
Change-Id: Ia716f73dffa94c9fecbcc1a7dd0f10cb62639671
This patch removes the compatibility with OVN under v20.09. That
implies the OVN Southbound definition has "Chassis_Private" table.
Any previous check is removed from the code.
This patch also adds a sanity check, testing that the OVN Southbound
database definition is greater or equal to 2.9.0 [1].
The testing OVN NB and SB schemas are updated to the files contained in
OVN v22.09. The new testing NB schema version is 6.3.9; the new testing
SB schema version is 20.25.0.
[1]4adc10f581
Closes-Bug: #2002839
Change-Id: If64c967b89099946165bfaf66247def4881af832
This patch reverts [1], that was inherited during the ML2/OVN
migration to this repository. The Neutron functional CI is
stable without this workaround.
Related-Bug: #1808146
Change-Id: I466e4369374f0cf2b59fd5145f5a9b2dc5aaaa51
Since OVN 20.06 [1], the OVN configuration is stored in
"Chassis.other_config".
Since OVN 22.09, the "Chassis" configuration stored in
"Chassis.other_config" will not be replicated to
"Chassis.external_ids".
The ML2/OVN plugin tries to retrieve the "Chassis"
configuration from the "other_config" field first; if this
field does not exist (in OVN versions before 20.06), the plugin
will use "external_ids" field instead. Neutron will be compatible
with the different OVN versions (with and without "other_config"
field).
[1]74d90c2223
[2]51309429cc
NOTE: this patch is similar to [1], but in this case neutron keeps
compatibility with the different OVN versions (with and without
"other_config" field). Since [2], the Neutron CI has a new job that
uses the OVN/OVS packages distributed by the operating system
installed by the CI (in this case, Ubuntu 20.04 and OVN 20.03).
[1]https://review.opendev.org/c/openstack/neutron/+/859642
[2]https://review.opendev.org/c/openstack/neutron/+/860636
Closes-Bug: #1990229
Change-Id: I54c8fd4d065ae537f396408df16832b158ee8998
This reverts commit f8fa909444.
Reason for revert:
Because the OpenStack CI OS (Ubuntu 20.04) still doesn't support OVN > v20.03, we need to revert this patch until the CI migrates to Ubuntu 22.04
Change-Id: Iafd0ce629bf3a93454f4e322780d3b2f911363f8
Closes-Bug: #1991962
Since OVN 20.06 [1], the OVN configuration is stored in
"Chassis.other_config".
Since OVN 22.09, the "Chassis" configuration stored in
"Chassis.other_config" will not be replicated to
"Chassis.external_ids".
This patch replaces any reference to "external_ids" when
retrieving the Chassis configuration. That means Neutron
no longer supports OVN versions below 20.06.
[1]74d90c2223
[2]51309429cc
Closes-Bug: #1990229
Change-Id: If379cefb66a262318858e1cea6503aa3a56ba956
This reverts commit 819a1bb3e6.
Reason for revert:
This patch does not take care of other controller nodes registers.
We need to revert it to find a better solution.
Change-Id: I5a547c6baab8e5307a531480b0e10859256ad25d
The "ovn_hash_ring" procedure to clean up the stale/old registers
is now executed on the ``HashRingHealthCheckPeriodics`` class, tha
is executed on the ``MaintenanceWorker`` process.
In a HA scenario, if several servers are rebooted at the same time,
the "ovn_hash_ring" clean up operation can clash with API worker
method "_load_hash_ring", that executed a SQL read from this table.
In some high loaded environments, if the OVN database takes time
to be locally cached, this read operation is executed thousand of
times; basically any time an OVN database event occurs.
In order to avoid/skip a deadlock when deleting the "ovn_hash_ring"
table, this clean up is executed in a periodic task. If this task
succeeds, the task is stopped. If the task raises a database
exception, it is processed again.
Now the "ovn_hash_ring" registers are retrieved using the
"created_at" time as a filter. The initial time is taken when the
OVN mechanism driver is initilized, before any API worker is spawned
and any new "ovn_hash_ring" register has been created (an API
worker, when started, will create a new "ovn_hash_ring" register).
Any stale/old register stored in this table will be ignored; that
means any register created before the OVN mechanism driver was
started.
Closes-Bug: #1990174
Change-Id: I07c4cb6e20b8a84e4ace7a8e34555aced5b5da9f
Since [1], the "external_ids" of the NB Logical_Router_Port register
are copied into the SB Port_Binding "external_ids". When a change
in a Port_Binding register is received, if only the
"external_ids:revision_number" is changed, we skip any update on the
related Logical_Router_Port.
If not, that will lead to an infinite loop: Neutron will update
the Logical_Router_Port with a new revision number and OVN will
copy this new revision number to the SB register, triggering again
the update of the NB Logical_Router_Port
[1]https://www.mail-archive.com/ovs-dev@openvswitch.org/msg62836.html
Closes-Bug: #1973347
Change-Id: Ib51764778a666050c42de0dfeb9bf9b185d44bb7
nb_cfg_timestamp: The timestamp when ovn-controller finishes
processing the change corresponding to nb_cfg[1]. it can better
reflect the status of chassis.
This patch updated some unit tests. ensure mock 'time.time' is
stopped after test. if not stop, may affect "timeutils.utcnow_ts"
to obtain the real time, cause test case
'test_agent_with_nb_cfg_timestamp_not_timeout' failure.
Partial-bug: #1938478
[1] https://www.ovn.org/support/dist-docs/ovn-sb.5.html
Change-Id: Ia74a9404411862dc88b48c4a198d5c53f5f52704
Continue similar approach following in [1], where some project imports
collide with config options.
As part of the change, a wrapped decorator has been implemented to cover
those functions that include any of the ovn config options as value to
the decorators arguments (e.g. tenacity retry). This way we avoid
requiring the options to be registered as soon as the module is imported,
where they have not yet been registered by a main process.
[1] https://review.opendev.org/c/openstack/neutron/+/837392
Co-authored-by: Jakub Libosvar <libosvar@redhat.com>
Co-authored-by: Fernando Royo <froyo@redhat.com>
Change-Id: I4bccb094ee7f690cbc352c38b5b39d505e6ea460
Each functional test method create new empty NB and SB DBs. Each DB
has its own table NB_Global or SB_Global respectively that contains
exactly one record created by ovn-northd. When functional test spawns
northd, it populates the DBs however it doesn't guarantee the running
test in parallel will not attempt to access the record.
This patch makes sure that northd creates the records before it moves on
and considers northd spawned.
Closes-Bug: #1952004
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Change-Id: Ic936864aebcabb811860e17913fbff311c52845f
when chassis's available zone updated, triger rescheduler router
gateway chassis.
Closes-bug: #1958225
Change-Id: If7cf55f2c12b388fc34fa942b6b0bf70cb9eb866
Added a new OVN Client extension: OVNClientPlacementExtension. This
extension is in charge of handling the bandwidth information stored
in the OVN database, in the "Chassis" registers on the
"ovn-cms-options" dictionary.
Three new keys are created to store the resource provider information
needed to parameterize the network backend bandwidth information,
following the implementation done in OVS and SR-IOV:
- resource_provider_bandwidths
- resource_provider_inventory_defaults
- resource_provider_hypervisors
When the OVN Client is started, the Placement extension will check if
the "placement" extension is loaded. It will also create an event to
check any configuration change done in any "Chassis" register.
The Placement extension will read the initial configuration stored
in the OVN database and will populate it to Placement API, creating
the needed resource providers, traits and inventories.
NOTE: This patch belongs to a series of patches to implement
minimum bandwidth scheduling blueprint in OVN backend. The next
patch will make OVN backend scheduling aware using the information
stored in Placement API and the port information passed by Nova when
a VM is created.
NOTE: this patch improves [1], fixing the error [2] reported in [3].
[1]https://review.opendev.org/c/openstack/neutron/+/776701
[2]https://paste.opendev.org/show/807614/
[3]https://launchpad.net/bugs/1936983
Partial-Bug: #1578989
Change-Id: I63e81aebce2621226ff2cfe91f16c97913c137e8
This adds support to test_metadata_agent tests for OVN versions
that have the Chassis_Private table.
Closes-Bug: 1944460
Change-Id: If4e24dd933c0efbb793ed6082c59db0435833389
Also updates the docs to be clearer on OVN-Geneve relation topics.
Co-Authored-By: Elvira García Ruiz <egarciar@redhat.com>
Change-Id: Ia253cc4d85261ce1535f4d27b3da91275d879903
Closes-bug: #1868137
This reverts commit df5cb28737.
The reverted commit triggers the failure of tempest-slow-py3 job.
tempest-slow-py3 is equal to non-voting neutron-ovn-tempest-slow job
in the neutron CI. It is a non-voting job so the error was not detected
before merging it. To recover the tempest job in OpenStack wide,
this commit reverts it.
See http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023764.html
Closes-Bug: #1936983
Change-Id: Id8cdd9c69e4fef2d9c335447b498958add8b7816
Added a new OVN Client extension: OVNClientPlacementExtension. This
extension is in charge of handling the bandwidth information stored
in the OVN database, in the "Chassis" registers on the
"ovn-cms-options" dictionary.
Three new keys are created to store the resource provider information
needed to parameterize the network backend bandwidth information,
following the implementation done in OVS and SR-IOV:
- resource_provider_bandwidths
- resource_provider_inventory_defaults
- resource_provider_hypervisors
When the OVN Client is started, the Placement extension will check if
the "placement" extension is loaded. It will also create an event to
check any configuration change done in any "Chassis" register.
The Placement extension will read the initial configuration stored
in the OVN database and will populate it to Placement API, creating
the needed resource providers, traits and inventories.
NOTE: This patch belongs to a series of patches to implement
minimum bandwidth scheduling blueprint in OVN backend. The next
patch will make OVN backend scheduling aware using the information
stored in Placement API and the port information passed by Nova when
a VM is created.
Partial-Bug: #1578989
Change-Id: I8ba38b8ace8852009fba8712aafa9f88c2b93ccb
It is possible for events from the nb/sb to fire before the opposite
db connection is made. These events can call back into driver code
which tries to access the other db before it is connected.
Closes-Bug: #1929197
Closes-Bug: #1928794
Closes-Bug: #1929633
Change-Id: If947581b90ced42981c4611c32de8f428a052c69
The functional tests collected OVN DBs logs but didn't collect northd
logs that may be also useful for troubleshooting test errors.
Change-Id: Iea642ded6f7b9a6b24870dee9808b74a5e765b0e
"OVNMechanismDriver._clean_hash_ring" is called at exit. Because the
SQL connection is already closed, an exception is logged. This patch
mocks this method and avoids the OVN FTs to log this message.
Closes-Bug: #1919352
Change-Id: I6c68d15d0af644ab00467bea8e0cd3ca816f5cc5
This is patchset 2 of 2 for OVN driver handling of security-group-logging.
It includes the core changes and tests for this feature.
This feature requires OVN 20.12 [0] or newer. Functional test will be
skipped for non-supported versions.
Related-Bug: 1468366
Closes-Bug: 1914757
[0]: 880dca99ea
Change-Id: Ic86fa70eb34c9b178267b80de1f8883a3ef03e98
Signed-off-by: Flavio Fernandes <flaviof@redhat.com>
Classes MaintenanceThread and HashRingHealthCheckPeriodics
are now mocked in the TestOVNFunctionalBase.
Closes-Bug: #1917370
Change-Id: Ie91a6a50fbda806f0c6ccafa9d3b38083715df65
96eae957b0 changed initialization for base
tests but logging configuration gets lost after super().setUp()
Revert this part as it is not needed in the end for tests to pass
properly
Change-Id: I4a8ca76920b0f174b37171c09653b6755a2dda5c
Closes-Bug: #1915272
Reset "AgentCache" singleton instance after each functional test
execution. That will remove any stored information from the previous
tests.
Related-Bug: #1914754
Change-Id: Ibd8897ec04843a5d6eb8639cc0c6ef0e2b9d2a3b
If the Chassis_Private table exists in the schema, add_fake_chassis
will need to create it since we aren't running ovn-controller.
Closes-Bug: #1914754
Change-Id: Ie36cb025540c158fbbed4126be5d654888715300
get_schema_helper() makes a direct short-lived connection to the
ovsb-server to grab the running schema before we initiate a
connection for the Idl. Moving the schema retrieving code out of
the path of starting the connection allows us to adjust to the
ovsdb-server's schema version at driver init instead of having to
set it in post_fork_initialize().
One benefit of this is that Events can be more easily defined to
watch for a table which might change depending on the schema, e.g.
Chassis vs Chassis_Private.
Change-Id: Ia3dd792f4335fcde0df7d11e6aaf693dd9611bb6
The current code looks at a hash ring node's created_at/updated_at
fields and tries to determine whether the node has been updated
based on whether updated_at - created_at > 1 second (due to the
method that initially fills them being different by microseconds).
Unfortunately, due to the notify() method being called which calls
the hash ring node's touch_node(), a node can be updated in under
a second, meaning we will prevent caching for much longer than
we intend.
When using sqlite in-memory db, this continually re-creating the
Hash Ring objects for every event that is processed is exposing an
issue where rows that should be in the db just *aren't*.
This patch instead limits the hash ring nodes to api workers and
prevents caching only until the number of nodes == number of api
workers on the host. The switch from spawning hash ring nodes
where !is_maintenance to is_api_worker is primarily because it
seems to be difficult to get a list of *all* workers from which to
subtract the maintenance worker so that _wait_startup_before_caching
can wait for that specific number of workers. In practice, this
means that RpcWorker and ServiceWorker workers would not process
HashRing events.
A note on bug 1903008: While this change will greatly reduce the
likelihood of this issue taking place, we still have some work to
do in order to fully understand why it rubs the database backend
in the wrong way. Thus, we will make this change 'related to'
instead of closing the bug.
Related-Bug: #1894117
Related-Bug: #1903008
Change-Id: Ia198d45f49bddda549a0e70a3374b8339f88887b
This is a subset of the changes for implementing the floating IP
port forwarding feature in neutron, using OVN as the backend.
This changeset covers the core implementation for portforwarding/drivers/ovn,
mech_driver, ovn-router as well as a subset of tests.
Port forwarding support in ovn_db_sync is not included here to facilitate review.
That, as well as all other supporting changes, are under the ovn/port_forwarding topic:
https://review.opendev.org/#/q/topic:ovn/port_forwarding+(status:open+OR+status:merged)
Depends-On: https://review.opendev.org/#/c/726478/
Partially-implements: ovn/port_forwarding
Partial-Bug: #1877447
Change-Id: I019fe11ac1ddcf2304f3f144c62d52667fc11dce
If new segment is created/old deleted we should update its
localnet port in related Logical_Switch.
Added also missing code to sync tool in order to delete provnet
ports in case of leftovers.
Change-Id: I6b864ba1c168643640a64bd7c25e1d0fc0ea348a
Related-Bug: 1865889
We still can see occasional failures because of timeouts
in the test_ovn_db_sync tests, like 1 failure per week.
As per agreement during last IRC meeting we decided to
bump up the functional tests timout from 15 seconds to 30 seconds,
because we're still pretty behind the default value of 180 seconds.
Change-Id: Ib20cdd0bb7d24795c8bd5c84c6143000b9922b4d
Related-Bug: 1868110
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This completes removal of all references to "import mock",
changing to "from unittest import mock" in fullstack and
functional tests.
Added a hacking check to enforce it in future patches.
Change-Id: Ifcaf1c21bea0ec3c35278e49cecc90a101a82113
The patch adds a short living connection in pre-fork routine that
creates neutron_pg_drop Port Group. Later after workers are spawned,
each worker also creates a short living connection and waits for an
event that the Port Group was created.
The short living IDLs limit its tables only for relevant tables so it
doesn't fetch the whole OVS DB to the local copy.
Closes-bug: #1866068
Change-Id: I1f5af36b8c3d5650f890edfed3c33dc206869824
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
On heavy loaded environments, like Neutron gates, we can
observe sporadic failures of functional tests, that are
timeouts.
Lets increase the timeout value to 15 seconds for functional
tests because looks like 5 seconds is not enought.
Change-Id: I327de751e3ba26c5be03b2571b105492661999cb
Closes-Bug: 1868110
Frode Nordahl