Extend the get_hosts_mapped_with_segments method to add
optional filters to include/exclude based on agent type.
Uses a joined query, when both include and exclude
filtering is used togheter the exclude filter is most
significant.
Partial-Bug: #2040172
Change-Id: I2cfd52a2657fad989e24e974fda470ecd960262b
Signed-off-by: Harald Jensås <hjensas@redhat.com>
Method ``_add_segment_host_mapping_for_segment`` is called by the event
(resources.SEGMENT, events.PRECOMMIT_CREATE), from
``SegmentDbMixin._create_segment_db``, and is called inside a database
writer context. That means it is irrelevant to check if the session is
active (must be always).
Closes-Bug: #1975542
Change-Id: Ib19dacf886486876237ed1157fb95ae157ed430e
This patch adds debug messages when a segment (or a set of
segments) is mapped to a host (or a set of hosts). The
deletion of an entry is logged too.
Trivial-Fix
Change-Id: I614dc6b8da9596b748db285709f9fbb146560b6f
It is available in Neutron lib since version 1.16 so pretty long time
now.
Also use segment api definition from neutron-lib, it's available
since version 1.19.0. The api definition from neutron-lib also
avoids circulary dependency b/w standard-attr-segment and segment
extension[1].
[1] https://review.opendev.org/c/openstack/neutron-lib/+/577866
Change-Id: I13699f8c494a15d8bb9e13f767f2725f7cab9f4f
Related-Bug: #1765008
When:
* the segments service plugin is enabled and
* we have multiple rpc worker processes (as in the sum of rpc_workers
and rpc_state_report_workers, since both kind processes agent
state_reports) and
* many ovs-agents report physnets,
then rabbitmq dispatches the state_report messages between the workers
in a round robin fashion, therefore eventually the state_reports of the
same agent will hit all rpc workers.
Unfortunately all worker processes have a 'reported_hosts' set to
remember from which host it has seen agent reports already. But right
after a server start when that set is still empty, each worker will
unconditionally write the received physnet-segment information into
the db. This means we multiply the load on the db and rpc workers by
a factor of the rpc worker count.
This patch tries to reduce the load on the db by adding another early
return before the unconditional db write.
Change-Id: I935186b6ee95f0cae8dc05869d9742c8fb3353c3
Closes-Bug: #1952730
This patch switches over to the payload style of callbacks for
NETWORK based events. As part of this change a few shims are needed
to handle cases where some callbacks don't yet use payloads and others
do. Once we move over to payloads for all callbacks the shims can be
removed.
NeutronLibImpact
Change-Id: I889364b5d184d47a79fe6ed604ce13a4b334acfa
This patch switches over to payload style of callbacks for the remaining
SEGMENT based callback events. Unit tests are also updated as needed
to handle the switch.
NeutronLibImpact
Change-Id: I3624228238babc64fe02bf0cc56eb0c1f6d35e73
- pass existing network dict to PRECOMMIT_DELETE ml2 handler;
- skip MTU and segment update handling in case network is
about to be deleted.
Above gives up to 50% net delete speed-up
Change-Id: I07c70db027f2ae03ffb5a95072e019e8a5fdc411
In case when enforce_new_defaults is set to True and new policy rules
are used, context.is_admin flag isn't really working as it was with old
rules.
But in case when elevated context is needed, it means that we need
context which has full rights to the system. So we should also set
"system_scope" parameter to "all" to be sure that system scope queries
can be done with such elevated context always.
It is needed e.g. when elevated context is used to get some data from
db. In such case we need to have db query which will not be scoped to
the single project_id and with new defaults to achieve that system_scope
has to be set to "all".
Proper fix for that should be done in neutron-lib and it is proposed
in [1] already but as we are have frozen neutron-lib version for
stable/wallaby already this patch for neutron is temporary fix for that
issue.
We can revert that patch as soon as we will be in Xena development cycle
and [1] will be merged and released.
[1] https://review.opendev.org/c/openstack/neutron-lib/+/781625
Related-Bug: #1920001
Change-Id: I0068c1de09f5c6fae5bb5cd0d6f26f451e701939
When a network is deleted, only one Placement call per segment
is done to remove the associated (if existing) resource provider.
Before this patch, each time a subnet was deleted, the segment
resource provider was updated. When no subnets were present in the
related segment, the associated resource provider was deleted.
This optimization improves the network deletion time (see Launchpad
bug). E.g.: a network with two segments and ten subnets, the Neutron
server processing time dropped from 8.2 seconds to 4.4 seconds (note
that the poor performance was due to the modest testing environment).
Along with the segment RP optimization during the network deletion,
this patch also skips the router subnet update. Because all subnets
in the network are going to be deleted, there is no need to update
them during the network deletion process.
Change-Id: Ifd50027911a9ca3508e80e0de9a6cc45b67006cf
Closes-Bug: #1878916
This patch switches over to payload style callbacks for all
SEGMENT_HOST_MAPPING events.
NeutronLibImpact
Change-Id: I71888b2b72b486c32991d651cdc608adb2149500
The functionality within neutron.db.common_db_mixin is available via
neutron-lib APIs. This patch removes common_db_mixin and updates any
uses of it to use neutron-lib instead.
Depends-On: https://review.openstack.org/#/c/636159/
NeutronLibImpact
Change-Id: I2388f90b37abb09408809dda8c21da551bcd94bb
The _resource_extend module is already rehomed into neutron-lib and is
shimmed in neutron. This patch removes the module as no active
consumers are using it.
NeutronLibImpact
Change-Id: I1550075fa5fa2aa2f1a88ee7189d311a1fe78391
This patch switches callbacks over to the payload object style events
for AGENT AFTER_CREATE and AFTER_UPDATE based notifications. To do
so a DBEventPayload object is used with the publish() method to
pass along the API related data.
Change-Id: Ibefa495be41c91957c2e8d797130e569bccc3765
This patch adds a check to determine if the 'segments' service plugin is
enabled. The segment-host mapping db table should only be saved and
updated to the db table if the users configure the 'segments' service
plugin in the config file. The data should be available only in a routed
network resource situation.
Change-Id: I65a42aa2129bef696906a18d82575461dc02ba21
Closes-Bug: #1799328
This patch switches over to the payload style of callbacks for SEGMENT
BEFORE_DELETE events. It does so by using the publish callback function
with DBEventPayload payloads. Corresponding listener callbacks are
updated where necessary.
NeutronLibImpact
Change-Id: I64031129ad458d4468a316a9385288ced091d333
Make the same status information available to notification consumers
(resource==agent, event==after_create/after_update) as it was already
available where the notification is sent in class AgentDbMixin.
Change-Id: Ie74091da934c7e49fd29ae4c6f930a7eb47e14b2
Partial-Bug: #1578989
See-Also: https://review.openstack.org/502306 (nova spec)
See-Also: https://review.openstack.org/508149 (neutron spec)
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.
NeutronLibImpact
Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
The retry_if_session_inactive decorator was rehomed into neutron-lib
[1]. This patch consumes it by removing the function from neutron and
using neutron-libs version where appropriate.
NeutronLibImpact
[1] https://review.openstack.org/#/c/557040/
Change-Id: I3e3289f33e62d45933d0fbf165bb4b25078f22d5
This incorporates flake8 2.6.x and pycodestyle will be used
instead of older pep8. This ensures future python3 compatibility
and a bit better code styling.
Change-Id: Ia7c7c5a44727f615a151e1e68dd94c7ed42f974f
The APIs our consumers are using from neutron.db_utils were rehomed into
neutron-lib with https://review.openstack.org/#/c/540161/
This patch consumes them by removing the rehomed APIs and using lib's
implementation where applicable.
NeutronLibImpact
Change-Id: I7ee53bce917feae8e37bf278eb3121a5af47131c
Standard attributes (i.e. revision_number, created_at, updated_at)
are missing in the segment resource. This is because the controller
doesn't process extensions on rendering the response. This patch
fixes it. In particular, this patch does the following:
* Process extensions when making the segment dict. This allows
other extensions to add attributes to segment response.
* Make segment extension implement update_attributes_map.
Tempest test: https://review.openstack.org/#/c/558609/
APIImpact update api reference of segments
Closes-Bug: #1760902
Change-Id: I3c93818002c2d7753454547231ba08544b6fa1c0
The segment extension's API definition was rehomed into neutron-lib with
commit I8b598c9aff2f0b28e5b11d3ab3570aeae2df1a4b
This patch consumes it by using lib's version of the module, removing
the rehomed code and updating the segment extension's test to mock
out standard_attr to ensure the description of segments isn't
overridden by the standardattrdescription extension.
NeutronLibImpact
Change-Id: Icea5981a5b920a30e159a0271052b9cc43236bc5
When multiple agents register at the same time there is
a possible race condition causing segment host mappings
updates to fail. StaleDataError raised by SQLAlchemy ORM.
Adding retry_if_session_inactive() decorator to the method
fixes the issue.
Also serialize the method with lockutils. It takes 25+
seconds to update segment host mappings for 10 agents with
the retry decorator alone. With the method serialized the
same operation completes in less than 1 second. The retry
decorator is still required for active/active scenarios.
Closes-Bug: #1743579
Change-Id: I616457f094d000a4016c610b454be8269d9b4948
This essentially reverts commit 12d24abba7.
Making the callback _delete_segments_for_network respond to
BEFORE_DELETE network event has created some bugs. In one of them,
it is not possible to delete a routed network, because the segments
cannot be deleted due to the fact that the associated subnets still
exist.
Making _delete_segments_for_network respond to PRECOMMIT_DELETE
introduces a StaleDataError with the standard attributes of the
deleted segments. To work around that, network_db is expired and
read again after notifying the PRECOMMIT_DELETE event in
delete_network in the DB core plug-in.
This also fixes an issue where we could delete the segment ID
of the l3-ha network when deleting a router, leaving all other
routers non-functioning. Moving this to PRECOMMIT_DELETE fixes
it since it is done after we have checked that the network is
not in use and can be deleted.
Closes-Bug: #1697324
Closes-Bug: #1732543
Change-Id: I7c3c4654f183b317647a28d599a538fe460db68f
The segmentation_id of a new segment is assigned by the
_handle_segment_change callback in the ML2 plugin, which is executed
after the segment was created, before commiting it to the DB. This
patchset adds code to update the newly created segment with the
segmentation_id assigned by the callback.
Change-Id: I493278a0bf5a3a0aadad10e5bee546d83b949fdc
Closes-Bug: #1698596
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
This gets rid of the last part of the segments module referencing the
model directly.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Change-Id: I9135aa7a3e952340654d3de51bf11a9bec71cbf2
get_objects doesn't accept dict_keys objects that sometimes may be
passed in by consumers of the code (for example, ovn does just that).
To make sure we still work for those consumers without a change on their
side, enforce conversion to a list before passing the list of physical
network names down the stack.
Depends-On: Ied0516686e86167a0c7b1d480eb1db2789f7cada
Change-Id: Ic6c5220cf3f1e901e159e0b76534155985ce9fe2
Usage reader and writer for db operations.
In Neutron code we have common situation like:
with context.session.begin():
context.session.add(obj)
self._make_obj_dict(obj)
With new enginefacade we change context.session.begin() for
db.context_manager.writer(reader).using(context).
When object leaves this with-block, its reference to session is
cleared because session is discarded. To use this object later to
load some data from its dependencies, we have to provide different
session to work in. To solve this obj either can be moved under
with-block or we have to do context.session.add(obj) one more time
to be able to load relations.
This change also switches to usage of new enginefacade for some db
operations with ports, in order to pass unit and functional tests.
Partially-Implements blueprint: enginefacade-switch
Change-Id: Ia15c63f94d2c67791da3b65546e59f6929c8c685
Deleting segments results in BEFORE/PRECOMMIT/AFTER events
itself so it shouldn't performed while already inside a transaction
to delete a network.
Closes-Bug: #1675910
Change-Id: I6c8070e1dc2216ea909288ce1133321e8eda1daa
Those were moved under neutron.db.models start of Ocata. We should be
able to clean them up now.
NeutronLibImpact
Change-Id: Iadbf44d52ee8e30712807384152a29ce1a8b8f72
These checks aren't necessary when triggered as part of a
network delete operation since any remaining subnets and ports
will be removed from the network. Additionally, they can race
with the auto cleanup methods for ports and subnets if segment
deletion is moved to BEFORE_DELETE where it should be.
Change-Id: I996f5b22f5179f5540760cc05c9f091bb3f3069b
Using Nova's generic resource pools (GRP) API, publish routed networks
IPv4 inventory. This inventory is then used by the Nova scheduler to
place instances based on the availability of IPv4 addresses in
routed networks segments.
Change-Id: Ib6b00c4889d6a34765844ce46280819dff0108c5
Partially-Implements: blueprint routed-networks
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Extract all the common utils from common_db_mixin.py in preparation
for moving them to neutron-lib.
This is a preliminary step in preparation for refactoring the
CommonDbMixin class and moving it to neutron-lib also.
Partial Blueprint: neutron-lib
Change-Id: I3cba375a8162cb68e8f988f22f5c8b1ce7915180
This patch introduces and integrates OVO for SegmentHostMapping.
Change-Id: I99598cf6fa4aefe7d3faee5cb0867d8ea1fff5c2
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Harald Jensås