The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
NeutronLibImpact
Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
The common rpc and exceptions were rehomed into
neutron-lib with [1]. This patch shims those rehomed
modules in neutron to switch over to neutron-lib's
versions under the covers.
To do so:
- The rpc and common exceptions are changed to
reference their counterpart in neutron-lib effectively
swapping the impl over to neutron-lib.
- The fake_notifier is removed from neutron and lib's
version is used instead.
- The rpc tests are removed; they live in lib now.
- A few unit test related changes are required
including changing mock.patch to mock.patch.object,
changing the mock checks for a few UTs as they don't
quite work the same with the shim in place.
- Using the RPC fixture from neutron-lib rather than
that setup in neutron's base test class.
With this shim in place, consumers are effectively using
neutron-lib's RPC plumbing and thus we can move consumers
over to neutron-lib's version at will. Once all
consumers are moved over we can come back and remove
the RPC logic from neutron and follow-up with a consumption
patch.
NeutronLibImpact
[1] https://review.openstack.org/#/c/319328/
Change-Id: I87685be8764a152ac24366f13e190de9d4f6f8d8
The neutron.common.rpc.create_connection function is just a reference to
the Connection class constructor. This patch removes create_connection
and replaces all uses with Connection instead.
NeutronLibImpact
Change-Id: I2f4b24ba732be47fc9911be1e24406fb1ffe821e
This patch is a fix for the issue B101 assert_used found by bandit
scanner:
https://docs.openstack.org/bandit/latest/plugins/b101_assert_used.html
As assert functionality could be turned off by the -O option for python
interpreter, assert statements were replaced to raise AssertionError.
The main reasoning for that is rest code after assert could be not
ready for values that assert filters.
Change-Id: I92a871b1b496c96c0c76cb37e4dda51bfc007e38
Related-Bug: #1759250
Today neutron.common.rpc has a handful of public constants and functions
that are not used outside of neutron. This patch removes those not
externally used and enhances the rpc.init function to accept extra
RPC exception modules (though it's not used today) to still support
the notion of "extra mods" for RPC.
This patch also paves the way for easier rehoming the module to
neutron-lib.
Change-Id: I51607216a39980c17807dcccace3ab66b549607b
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
This helps catch a number of potential issues with the neutron
code in advance. A false positive can be skipped with #nosec
at the offending line, just like #noqa can be added on
offending module imports.
Change-Id: I3e4cbc94539dd2cce61bfa5cd0265e75d7336311
In upgrade scenario, this is a regular case (no agents created the
exchange, so fanout is ineffective), so logging it using ERROR level is
misleading and against guidelines.
Change-Id: If098071c9c19490af7eb0863dbab83e488019bd7
Related-Bug: #1705351
I15cc2d6ae48e505c2da121880e27481dedf36d3b catches exceptions for
specific RPC endpoints affected by a recent change related to
push-notifications. There may be more changes like that in the future,
so instead of fixing them one by one, this patch consistently catches
exceptions from all cast calls for all RPC clients.
Change-Id: Ia7e6cd717758a9d5b18fe9cb07c55938f52040ce
Partial-Bug: #1705351
The old endpoint generates a deprecation warning:
neutron/common/rpc.py:61: DeprecationWarning: Using function/method
'oslo_messaging.transport.get_transport()' is deprecated: use
get_rpc_transport or get_notification_transport
TrivialFix
Change-Id: I51a3878dd0a50be2e4c4c93e6e7db4d330a22f23
With backoff client, setting .timeout property on it doesn't take any
effect. It means that starting from Mitaka, we broke
quitting_rpc_timeout option.
Now, when the TERM signal is received, we reset the dict capturing
per-method timeouts; and we cap waiting times by the value of the
option. This significantly reduces time needed for the agent to
gracefully shut down.
Change-Id: I2d86ed7a6f337395bfcfdb0698ec685cf384f172
Related-Bug: #1663458
This should get rid of the following deprecation message:
FutureWarning: The access_policy argument is changing its default value
to <class 'oslo_messaging.rpc.dispatcher.DefaultRPCAccessPolicy'> in
version '?', please update the code to explicitly set None as the value:
access_policy defaults to LegacyRPCAccessPolicy which exposes private
methods. Explicitly set access_policy to DefaultRPCAccessPolicy or
ExplicitRPCAccessPolicy.
BTW the previous attempt [1] to get rid of the message was ineffective.
That's because the library does not distinguish between access_policy
explicitly picked by method consumer, and access_policy using the
default keyword value.
NeuttronLibImpact this change will break plugins that may expose RPC
endpoints that are prefixed with an underscore.
[1] Ia8922baf422ee84e55f05534678b2f0377843688
Change-Id: Ia2498eb5079050bfb4741c8b6c259bf1c116b8d6
oslo.messaging allow dispatcher to restrict endpoint methods since
5.11.0 in I42239e6c8a8be158ddf5c3b1773463b7dc93e881, set with
LegacyRPCAccessPolicy explicitly to ensure it's compatible and
fix FutureWarning like:
"The access_policy argument is changing its default value to <class
'oslo_messaging.rpc.dispatcher.DefaultRPCAccessPolicy'> in version '?',
please update the code to explicitly set None as the value:
access_policy defaults to LegacyRPCAccessPolicy which exposes private
methods. Explicitly set access_policy to DefaultRPCAccessPolicy or
ExplicitRPCAccessPolicy.
Change-Id: Ia8922baf422ee84e55f05534678b2f0377843688
Those are remnants from the oslo-incubator times. Also, oslo.messaging
deprecated [1] transport aliases since 5.2.0+ that is the minimal
version supported for stable/newton. The patch that bumped the minimal
version for Neutron landed 3 months+ ago, so we can proceed ripping
those aliases from the code base.
[1] I314cefa5fb1803fa7e21e3e34300e5ced31bba89
Change-Id: I9e66140ad0b10c50e216c78e4914c98313cada05
Closes-Bug: #1424728
The context objects accepts and ignores unknown keyword arguments. This
was to allow it to handle the deserialization of parameters it didn't
understand from the base class' to_dict method. This will make it
difficult to pass unknown attributes to the base class so fix the
from_dict method to only accept params it knows about.
Related-Bug: #1602081
Change-Id: Ic58a2025680e8e1ba4f8a177d898be457e2c3160
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
1) It checks that person who wants to trace is trusted and knows
secret HMAC key.
2) It starts tracing in case of proper trace headers
and adds first wsgi trace point, with info about HTTP request
* Add initialization of osprofiler at start of service
Currently that includes oslo.messaging notifer instance creation
to send Ceilometer backend notifications.
Neutron client change: Ic11796889075b2a0e589b70398fc4d4ed6f3ef7c
Co-authored-by: Ryan Moats <rmoats@us.ibm.com>
Depends-On: I5102eb46a7a377eca31375a0d64951ba1fdd035d
Closes-Bug: #1335640
DocImpact Add devref and operator documentation on how to use this
APIImpact
Change-Id: I7fa2ad57dc5763ce72cba6945ebcadef2188e8bd
This adds an exponential backoff mechanism for timeout values
on any RPC calls in Neutron that don't explicitly request a timeout
value. This will prevent the clients from DDoSing the server by
giving up on requests and retrying them before they are fulfilled.
Each RPC call method in each namespace gets its own timeout value since
some calls are expected to be much more expensive than others and we
don't want to modify the timeouts of cheap calls.
The backoff currently has no reduction mechanism under the assumption
that timeouts not legitimately caused by heavy system load
(i.e. messages completely dropped by AMQP) are rare enough that the
cost of shrinking the timeout back down and potentially causing
another server timeout isn't worth it. The timeout does have a ceiling
of 10 times the configured default timeout value.
Whenever a timeout exception occurs, the client will also sleep for a
random value between 0 and the configured default timeout value to
introduce a splay across all of the agents that may be trying to
communicate with the server.
This patch is intended to be uninvasive for candidacy to be
back-ported. A larger refactor of delivering data to the agents
is being discussed in I3af200ad84483e6e1fe619d516ff20bc87041f7c.
Closes-Bug: #1554332
Change-Id: I923e415c1b8e9a431be89221c78c14f39c42c80f
Commit 077e869fda marked the parameter
as unused and deprecated.
This patch removes the parameter.
Change-Id: I550c65861a8664068e340b7b4689b2345951e888
In oslo.messaging 2.9.0, the RPC and notification buses were decoupled
into separate transport. This changes over the Neutron notifier to use
the notification transport provided from oslo.messaging instead of the
original transport (which is now used for RPC). By default, the
notification transport will pull in the original transport if nothing is
configured in oslo_messaging_notification, so this can be an in-place
replacement without impacting upgrades or configurations.
This functionality was added to oslo.messaging to address bug #1504622
Change-Id: I89c8a84c81079677088cc7d656a0bcb5e323de4d
It's not used since the time we switched to oslo.messaging (Juno), it's
time to deprecate and eventually remove it.
Closes-Bug: #1506492
Change-Id: I57b0229c2b6028796cd10bbbfc9b166cf8a6dab0
Went through all of the docstrings in Neutron and did
some cleanup. I'm sure there are bunch more that I have missed.
Change-Id: Ib29d2de1c580880c89ed4fd069e1515d0977a3e7
This reverts commit 4f1fb05959.
Merge of master went in so this can be re-applied and have the
conflict resovled.
Conflicts:
neutron/server/__init__.py
Change-Id: I917e89b2191b3d4646e66a7a313783a56cf0ace5
This reverts commit 2ba2456f9d
because it has a conflict with upstream commit
9f6bd17703. We need to merge
master into pecan with this reverted and then revert the revert
and resolve the conflict there so we don't have any conflict
resolutions in merge commits.
Change-Id: I22388de076cbd90da33502249e101749e019e0cb
The Pecan WSGI server is supposed to server REST requests only.
As several plugins initialize their RPC interfaces upon initialization,
this results in establishing undesired connections to the AMQP server.
This patch does not alter plugin behavior or initialization process,
but ensures that the setup_rpc operation performed by a plugin has no
effect when executed in the Pecan WSGI server.
This patch also performs some refactoring for server launchers.
Change-Id: I56384f5f964ea90d72babf911aa5639989e9c3d8
oslo.service has graduated, so neutron should consume it.
Closes-Bug: #1466851
Depends-On: Ie0fd63f969f954029c3c3cf31337fbe38f59331a
Depends-On: I2093b37d411df9a26958fa50ff523c258bbe06ec
Depends-On: I4823d344878fc97e66ddd8fdae25c13a34dede40
Change-Id: I0155b3d8b72f6d031bf6f855488f80acebfc25d4
get_admin_roles was introduced so that contextes generated from
within plugins could be used for policy checks. This was the case
up to the Havana release as several plugins invoked the policy
engine directly to authorize requests.
This was an incorrect behaviour and has now been fixed, meaning
that get_admin_roles is no longer need and can be safely removed.
This will result in a leaner and more reliable codebase. Indeed the
function being removed here was the cause of several bugs where the
policy engine was initialized too early in the server bootstrap
process.
While this patch removes the feature it does not remove the
load_admin_roles parameter from context.get_admin_context. Doing so
will break other projects such as neutron-lbaas. The parameter is
deprecated by this patch and an appropriate warning emitted.
As a consequence neutron's will now no longer perform policy checks
when context.is_admin=True. This flag is instead set either when
a context is explicitly created for granting admin privileges, or
when Neutron is operating in noauth mode. In the latter case every
request is treated by neutron as an admin request, and get_admin_roles
is simply ensuring the appropriate roles get pushed into the context
so that the policy engine will grant admin rights to the request.
This behaviour is probably just a waste of resource; also it is not
adding anything from a security perspective.
On the other hand not performing checks when context.is_admin is
True should not pose a security threat either in noauth mode or
with the keystone middleware. In the former case the software keeps
operating assuming admin rights for every requests, whereas in the
latter case the keystone middleware will always supply a context
with the appropriate roles, and there is no way for an attacker
to trick keystonemiddleware into generating a context for which
is_admin=True.
Finally, this patch also does some non-trivial changes in test_l3.py
as some tests were mocking context.to_dict ignoring the is_admin flag.
Closes-Bug: #1446021
Change-Id: I8a5c02712a0b43f3e36a4f14620ebbd73fbfb03f
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
This patch adds missing Connection.close() which is referred from
Service.stop().
Closes bug 1413998
Change-Id: I76e825463012384f98a473dfda62665392e533f3
Oslo.messaging creates node topic (with node name as suffix)
and fanout topic automatically. There's no need to do this
in Neutron code.
Closes bug 1413156
Change-Id: Ic790cab648b213b55679038967fef12eb5cec606
There are a small number of examples of "eager" interpolation in
neutron:
logging.debug("foo %s" % arg)
These should be converted to perform the interpolation lazily within
the logging function, since if the severity is below the logging level
then the interpolation can be skipped entirely.
This change addresses all such current examples found in neutron core
via a pylint test. Vendor plugins and services are fixed elsewhere.
Change-Id: I823d8453cd76e4985cabd31ca6b939f43a80b36c
Partial-Bug: #1404788
All users of the RpcCallback class have been converted to directly
define the oslo.messaging Target so this compatibility class is no
longer needed.
This is the final change of blueprint drop-rpc-compat.
Change-Id: I25474569850f3f2c176e7eee7aafa7ff0180b809
All users of this class have now been converted to use oslo.messaging
APIs directly, so this compatibility class can be removed.
Part of blueprint drop-rpc-compat.
Change-Id: Ife5c96d2d737694b9e79fe079d62dc48f23c033a
Drop the last neutron.common.rpc exception mapping, MessagingHelper,
which wasn't actually used anywhere anyway.
Part of blueprint drop-rpc-compat.
Change-Id: I43b9b97accff9cc7684146076ba0b5d62302f06e
Remove RemoteError from neutron.common.rpc, which was just mapped
directly to oslo.messaging.RemoteError.
Part of blueprint drop-rpc-compat.
Change-Id: Ic4d7ea0adfd68eb56100e32ce0d8e59aa2aae5b1
Remove RPCException, which was just mapped directly to
oslo.messaging.MessagingException for the purposes of minimizing the
impact to the code base when moving from openstack.common.rpc to
oslo.messaging.
Part of blueprint drop-rpc-compat.
Change-Id: Idc5af503f24217fbd2d982c338434c200d025521
Factory method provided by the library should be used to create a server
object. This allows to change servers internal implementation without
touching end users.
Also removed custom dispatcher. Its only purpose was to log messages.
If logging is configured accordingly all incoming messages will be
logged from oslo/messaging/_drivers/amqpdriver.py [1]
[1] http://goo.gl/nV9tcu
Closes-Bug: #1357236
Change-Id: Ic208994c5a64fd48528cb41d30a975d68d84af05
RpcProxy class is used as base in classes with multiple inheritance
(ex. MeteringAgent). To initialize objects of such classes correctly
every constructor should call its base constructor via super().__init__()
Closes bug 1332788
Change-Id: I00ef804b6ee6c73dec3e177202bb85311e3e2b10
In Icehouse was used rpc library from oslo-incubator. Because the code
wasn't mature enough it was running in DEBUG mode. After migrating to
oslo.messaging we don't need to have debug level in oslo.messaging but
it's good to have communication logged.
Closes-Bug: #1340696
Change-Id: I4f68545053912f96affc1bbcd64fcd7efe8d18c0
Most of this code will probably stay with us for quite some time, so
let's make things easier and consider them as our way of doing RPC.
blueprint oslo-messaging
Change-Id: Iaf353b23f9c54b82d1e02a6bd5a5960cec827c88
oslo.messaging has a workaround [1] that requires context to be
transformed to pure dict before passing into amqpdriver.
Renamed serializer class to reflect its broader usage.
Updated FakeNotifier to expect serializer and other keyword arguments
supported by oslo.messaging Notifier class.
[1]: oslo/messaging/_drivers/amqpdriver.py#L337
blueprint oslo-messaging
Closes-Bug: #1332412
Change-Id: I7e7658d03639afae7bf6d3ad71445cb5b6459c09
Boden R