Remove B303 (md5, sha1 for python<3.9) and
remove B311 (Standard pseudo-random generators are
not suitable for security/cryptographic purpose) from
the skip list of bandit execution.
Change-Id: I6e9e61e7f94dc9ca339942529af8997adef45e38
rpc_workers can be set < 1 with 'ovn' backend when no
other agent is running apart from ovn agents to
consume these rpc notifications.
Add and apply disable_notifications decorator on
methods which do rpc cast calls to agents, the
decorator makes the caller method execute only
when rpc_workers >=1. This patch not changing
default behavior and utilizes the rpc_workers config option
to enable rpc notification on resources updates only when
rpc_workers >= 1.
Also set rpc_workers=0 in ovn jobs to cover this scenario.
Closes-Bug: #1889737
Closes-Bug: #1992352
Change-Id: I700fe2cd422bc1eb8b5144ec116e7f0a60238419
After port update, DHCP agent will be notified about changes only if
one of the port's attributes related somehow to the DHCP will change.
Such fields are:
* fixed_ips,
* MAC address,
* dns_domain,
* dns_name,
* dns_assignment,
* extra_dhcp_opts.
In other cases there is no reason to send notifications to the agent.
This will results with less notifications to the DHCP agent and less
possibilities to race condition between DHCP and L2 agents while
switching ports from the DOWN to ACTIVE status and sending notifications
to nova.
Closes-Bug: #1982367
Change-Id: If7990bdec435af76ad2e88fd4ea2bc24a255fd5a
This parameter, sent by the DHCP agent, is needed to remove the
workaround method "_get_network_lock_id".
The removal of this method will be done in [1] in Y release.
Related-Bug: #1732456
[1]https://review.opendev.org/c/openstack/neutron/+/800967
Change-Id: Ibd7fed33d314e901c69da33f42029f8ea67df98d
This patch switches over to callback payloads for PORT
AFTER_DELETE events.
Some shims were removed.
Change-Id: If69e37b84fe1b027777b1d673b3d08a6651a979e
This reverts commit 062336e59b.
Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.
Related-Bug: #1920001
Conflicts:
neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
neutron/common/utils.py
neutron/db/address_group_db.py
neutron/services/segments/db.py
Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
This patch switches the code over to the payload style of callbacks [1]
for PORT AFTER_CREATE events. In addition it adds a branch/shim to the
dhcp_rpc_agent_api to support both payload and kwarg style callbacks.
NeutronLibImpact
[1]
https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html
Change-Id: I25d43d4f8f2390b07e0d11c631f894d88669bbe0
This patch switches the code over to the payload style of callbacks [1]
for ROUTER_INTERFACE events for those that are not using them yet.
The unit tests are also updated where needed to account for the
payload style callbacks and publish() method. In addition, a few
callback methods that use the retry_if_session_inactive() decorator are
separated out from the callback so that the context can still be
passed and detected by retry_if_session_inactive logic.
NeutronLibImpact
[1]
https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html
Change-Id: I8d9f8296952dfb10fcccd6afd72e90a5d4f379eb
This patch switches over to the payload style of callbacks for
NETWORK based events. As part of this change a few shims are needed
to handle cases where some callbacks don't yet use payloads and others
do. Once we move over to payloads for all callbacks the shims can be
removed.
NeutronLibImpact
Change-Id: I889364b5d184d47a79fe6ed604ce13a4b334acfa
DHCP notification is done after each create/update/delete for
network, subnet and port.
This notification currently has to retrieve network from DB almost
every time, which is a quite heavy DB request and hence affects
performance of port and subnet CRUD.
This patch suggests 2 optimizations:
- do not fetch network if not needed (only fetch when schedule needed)
- for port and subnet AFTER_CREATE event pass network dict from plugin
According to Rally tests these changes improve performance:
- port create ~20%
- port update ~20%
- subnet create ~15%
- port delete and subnet update/delete - not tested
Closes-Bug: #1923161
Change-Id: I0ab836ac09225f4f3ad435e9ceaf315018855d52
In case when enforce_new_defaults is set to True and new policy rules
are used, context.is_admin flag isn't really working as it was with old
rules.
But in case when elevated context is needed, it means that we need
context which has full rights to the system. So we should also set
"system_scope" parameter to "all" to be sure that system scope queries
can be done with such elevated context always.
It is needed e.g. when elevated context is used to get some data from
db. In such case we need to have db query which will not be scoped to
the single project_id and with new defaults to achieve that system_scope
has to be set to "all".
Proper fix for that should be done in neutron-lib and it is proposed
in [1] already but as we are have frozen neutron-lib version for
stable/wallaby already this patch for neutron is temporary fix for that
issue.
We can revert that patch as soon as we will be in Xena development cycle
and [1] will be merged and released.
[1] https://review.opendev.org/c/openstack/neutron-lib/+/781625
Related-Bug: #1920001
Change-Id: I0068c1de09f5c6fae5bb5cd0d6f26f451e701939
When processing port events (create, update, delete), the port
provisioning (port creation) has priority over the other events [1].
As reported in the related bug, if a port deletion with an IP
address and another port creation with the same IP address arrive
to the DHCP agent, those events can be processed in the same queue.
Because of the creation event priority, even when this event arrived
after the deletion event, it will be processed first. That will
clash with the DHCP agent cache, that contains a port (not deleted
yet) with the same IP address. That will trigger an unwanted resync.
This patch implements a specific logic to store the events in
"ResourceProcessingQueue" (that uses "PriorityQueue" [2]). When
a port event arrives, the event comparison method checks the
(subnet, fixed_ips) tuple set of both elements. If there is a
coincidence, that means those ports are the same or are using
the same IP addreses (the race condition explained in the bug).
In this case, the priority is defined only by the timestamp;
that means the events are processed in order of arrival.
Because the Neutron server do not allow to have two ports in the
same subnet with the same IP address, the order of the events is
guaranteed. In the case explained in the bug, the deletion event
will be processed first.
[1]https://review.opendev.org/c/openstack/neutron/+/626830
[2]https://docs.python.org/3/library/queue.html#queue.PriorityQueue
Closes-Bug: #1913723
Change-Id: I89438feae3c0244f6da5e6a2a035d45b956ac247
Low port delete priority may lead to duplicate entries in network
cache if IPs are reused frequently.
Also can't find a strict reason why it should be of lower priority.
Change-Id: I55f858d50e636eb9091570b256380330b9ce9cb3
Related-bug: #1862315
Related-bug: #1828423
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
NeutronLibImpact
Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
Currently, the dhcp Provisioning of ports is the crucial bottleneck
of that concurrently boot multiple VM.
The root cause is that these ports will be processed one by one by dhcp
agent when they belong to the same network, And the 'Provisioning complete'
port is still blocked other port's processing in other dhcp agents. The
patch aim to optimize the dispatch strategy of the port cast to agent to
improve the Provisioning process.
In server side, I classify messages to multi levels. Especially, I classify
the port_update_end or port_create_end message to two levels, the high-level
message only cast to one agent, the low-level message cast to all agent. In
agent side I put these messages to `resource_processing_queue`, with the queue,
We can delete `_net_lock` and process these messages in order of priority.
Additonally, I modified the `resource_processing_queue` for my demand. I update
`_queue` from LIST to PriorityQueue in `ExclusiveResourceProcessor`, by this
way, we can sort all message which cached in `ExclusiveResourceProcessor` by
priority.
Related-Bug: #1760047
Change-Id: I255caa0571c42fb012fe882259ef181070beccef
The port delete events are not synchronized with network rpc events. This
creates a condition which makes it possible for a port delete event to be
processed just before a previously started network query completes.
The problematic order of operations is as follows:
1) a network is scheduled to an agent; a network rpc is sent to the
agent
2) the agent queries the network data from the server
3) while that query is in progress a port on that network is deleted; a
port rpc is sent to the agent
4) that port delete rpc is received before the network query rpc
completes
5) the port delete results in no action because the port was not present
on the agent
6) the network query finishes and adds the port to the cache (even
though the port has already been deleted)
7) some time passes and a new port is configured with the same IP
address as the port that was deleted in (3)
8) the dhcp host file is corrupted with 2 entries for the same IP
address.
9) dhcp queries for the newest port is rejected because of the duplicate
entry in the dhcp host file.
The solution is to add the network_id to the port_delete_end rpc event
so that the _net_lock(network_id) synchronization point can be acquired
so that it is processed serially with other network related events.
To ensure backwards compatibility with newer agents running against older
servers the determination of which network_id value to use in the lock is
handled using a utility that will fallback to the previous mode of operation
whenever the network_id attribute is not present in the *_delete_end RPC
events. That utility can be removed in the future when it is guaranteed
that the network_id attribute will be present in RPC messages from the
server.
Closes-Bug: #1732456
Change-Id: I735f8b1c9248b12e5feb6cbe970cf67f321e6ebc
Signed-off-by: Allain Legacy <allain.legacy@windriver.com>
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.
NeutronLibImpact
Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
The is_extension_supported function now lives in neutron-lib. This patch
removes the function from neutron and uses lib's version instead.
NeutronLibImpact
Change-Id: Iccb72e00f85043b3dff0299df7eb1279655e313e
This patch switches callbacks over to the payload object style events
[1] for BEFORE_RESPONSE and AFTER_REQUEST based notifications. To do
so an APIEventPayload object is used with the publish() method to
pass along the API related data. In addition a few UTs are updated to
work with the changes.
NeutronLibImpact
[1] https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html#event-payloads
Change-Id: Ibd8559e0db9dcc995abf8937a0cb764b21a18531
The neutron-lib commit I360545b6ee4291547e0c5c8e668ad03d3efa4725 moved
the externally consumed globals from neutron.common.constants into lib.
With the exception of PROVISIONAL_IPV6_PD_PREFIX all other constants
in neutron.common.constants should only be used in neutron, and will
hopefully remain that way. External consumers needing access to other
common constants should move them into lib first.
NeutronLibImpact
Change-Id: Ie4bcffccf626a6e1de84af01f3487feb825f8b65
Changing rpc_api.rst file path from doc/source/devref/rpc_api.rst
to /doc/source/contributor/internals/rpc_api.rst. Because rpc_api.rst
file is located at this path
doc/source/contributor/internals/rpc_api.rst.
Closes-Bug #1722072
Change-Id: Ic243aab9e3428bfec69db61a94b4129cd768e233
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
On profiling the get_devices_details communications between
the agent and the server, a significant amount of time
(60% in my dev env) is being spent in the AFTER_UPDATE events
for the port updates resulting from the port status changes.
One of the major offenders is the native DHCP agent notifier.
On each port update it ends up retrieving the network for the
port, the DHCP agents for the network, and the segments.
This patch addresses this particular issue by adding logic to
skip a DHCP notification if the only thing that changed on the
port was the status. The DHCP agent doesn't do anything based on
the status field so there is no need to update it when this is
the only change.
Change-Id: I948132924ec5021a9db78cf17efbba96b2500e8e
Partial-Bug: #1665215
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
This makes the notifier subscribe to core resource events
and leverage them if they are available. This solves the
issue where internal core plugin calls from service plugins
were not generating DHCP agent notifications.
Closes-Bug: #1621345
Change-Id: I607635601caff0322fd0c80c9023f5c4f663ca25
Bug 1591766 unveiled an issue where calling the plugin API does not trigger
DHCP notifications. This is required by the auto-allocated-topology service
plugin that calls core_plugin.update_network(), and expect notifications
to be sent out on state changes. To accomplish this, the logic has been
encapsulated in the DHCP module, and leveraged via callback mechanisms.
For this reason, new events have been introduced, AFTER_REQUEST, and
BEFORE_RESPONSE. The latter in particular is the one needed to hook up
dhcp notifications in order to preserve backward compatibility.
More precisely, core plugins that use DHCP as is or implement their own,
(with or without an agent) should already instantiate their own notifier,
and if they do not, this should be rectified.
A search on codesearch.openstack.org reveals that out-of-tree plugins
already specify their own notifiers, and the default initialization is
clearly redundant now.
Related-bug: #1591766
Change-Id: I7440becb6d30af7159ecaeba09d7a28eceb71bea
Since a network can now be broken up by segments, each segment will
need to have its own DHCP Agent. This maintains backwards
compatibility when a network does not have a segment. However,
once a segment is created on a network, a dhcp agent should be
scheduled per segment with a dhcp enabled subnet.
The scheduling happens by filtering the candidate dhcp agents by
the hosts that are bound to that segment.
Partially-Implements: blueprint routed-networks
Change-Id: If73211978e14b7533a1213cfb8c2c155a408f19e
In Dnsmasq, the function get_isolated_subnets() returns a list of
subnets in a network if the subnet is not connected to a router.
The implementation of this function checks all the router interface
ports in a cached network object passed from DHCP agent. But the
cached network object is not updated when a subnet is attached to
or detached from a router.
This patch fixes that by adding callback functions in DHCP RPC client
to notify DHCP agent when changes happen on router interfaces.
Closes-Bug: #1554825
Change-Id: Ifaab163f49e0d1c5cb3eba6efa96214104647e4e
Python 3 deprecated the logger.warn method, see:
https://docs.python.org/3/library/logging.html#logging.warning
so we prefer to use warning to avoid DeprecationWarning.
Closes-Bugs: #1529913
Change-Id: Icc01ce5fbd10880440cf75a2e0833394783464a0
Co-Authored-By: Gary Kotton <gkotton@vmware.com>
No reason to try get enabled_agents and send notification
if there are no any agents associated with network.
Closes-bug: #1522471
Change-Id: I111967415ce600253fc679837d03c9cd75f19656
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
This cannot be done in Python 3, where dict.keys() returns an iterator. We need
to cast the result of dict.keys() to a list first.
Change-Id: I28986aefb720b4513e3eee9ba0909f79d1dc9695
Blueprint: neutron-python3
Previously when admin_state_up of an agent is turned to False,
all services on it will be disabled.
This fix makes existing services on agents with admin_state_up
False keep available.
To keep current behavior available the following configuration
parameter added.
* enable_services_on_agents_with_admin_state_down
If the parameter is True, existing services on agents with admin_state_up
False keep available. No more service will be scheduled to the agent
automatically. But adding a service to the agent manually is available.
i.e. admin_state_up: False means to stop automatic scheduling under the
parameter is True.
The default of the parameter is False (current behavior).
Change-Id: Ifba606a5c1f3f07d717c7695a7a64e16238c2057
Closes-Bug: #1408488
Change the DHCP notifier behavior to schedule a network
to a DHCP agent when a subnet is created rather than
waiting for the first port to be created.
This will reduce the possibility to get a VM port created
and have it send a DHCP request before the DHCP agent is
ready. Before, the network would be scheduled to an agent
as a result of the API call to create the VM port, so the
DHCP port wouldn't be created until after the VM port.
After this patch, the network will have been scheduled to
a DHCP agent before the first VM port is created.
There is still a possibility that the DHCP agent could be
responding so slowly that it doesn't create its port and
activate the dnsmasq instance before the VM sends traffic.
A proper fix will ensure that the dnsmasq instance is
truly ready to serve requests for a new port will require
significantly more code for barriers (either on the subnet
creation, port creation, or the nova boot process) are too
complex to add this late in the cycle.
This patch also eliminates the logic in the n1kv plugin that
was already doing the same thing.
Closes-Bug: #1431105
Change-Id: I1c1caed0fdda6b801375a07f9252a9127058a07e
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
There is an rpc interface defined for the Neutron plugin to be able to
execute methods in the DHCP agent. Provide docstring pointers in the
client and server side that tells you where to find the other side of
the interface.
No namespace usage is needed here. This API is the only one exposed
via the DHCP agent, so the default namespace used now is fine.
The DhcpAgent class was updated to explicitly define the
messaging.Target(). Previously it was using the equivalent one
defined in the Manager base class. Having it specified here makes it
more obvious that this is an rpc endpoint, and also provides the
obvious place that must have the version updated if the interface is
changed.
Part of blueprint rpc-docs-and-namespaces.
Change-Id: I4a6eb8dacb9ba01f329a5d5961dc0e0ee6f780ba
Remove usage of the RpcProxy compatibility class from the
DhcpAgentNotifyAPI class. The equivalent oslo.messaging APIs are now
used instead.
Part of blueprint drop-rpc-compat.
Change-Id: Ib658a0d67da1af3b009bc6df9a7c8ec08c04897b
Mostly trivial import changes.
- oslo.i18n no longer provide install() method to inject _() into
globals(), so removed all calls to it;
- removed Babel from dependencies (it will now be grabbed by oslo.i18n);
- updated tox.ini to ignore import violations for oslo.i18n.
Change-Id: I6623d551f512fb7fe9bf35ee734ed6d4c6cbc287
elajkat
yatinkarel