The 'nova.exception_wrapper.wrap_exception' decorator accepted either a
pre-configured notifier or a 'get_notifier' function, but the forget was
never provided and the latter was consistently a notifier created via a
call to 'nova.rpc.get_notifier'. Simplify things by passing the
arguments relied by 'get_notifier' into 'wrap_exception', allowing the
latter to create the former for us.
While doing this rework, it became obvious that 'get_notifier' accepted
a 'published_id' that is never provided nowadays, so that is dropped. In
addition, a number of calls to 'get_notifier' were passing in
'host=CONF.host', which duplicated the default value for this parameter
and is therefore unnecessary. Finally, the unit tests are split up by
file, as they should be.
Change-Id: I89e1c13e8a0df18594593b1e80c60d177e0d9c4c
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Replace six.text_type with str.
A subsequent patch will replace other six.text_type.
Change-Id: I23bb9e539d08f5c6202909054c2dd49b6c7a7a0e
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
We're occasionally seeing stacktraces like this in our tests:
Fatal Python error: Cannot recover from stack overflow.
Current thread 0x00007fe66549f740 (most recent call first):
File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614 in _get
File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2183 in __getattr__
File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614 in _get
File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2183 in __getattr__
File "...nova/.tox/py36/lib/python3.6/site-packages/oslo_config/cfg.py", line 2614 in _get
...
From a look at the oslo.config source, this seems to be occurring
because 'ConfigOpts.__cache' is somehow undefined, which results in the
'_get' method attempting to call '__getattr__' [1], which calls '_get'
[2], which calls '__getattr__' and so on.
The exact reason this is happening isn't clear, but what is clear is
that how we handle global config options in the tests that are failing
is very strange and potentially subject to race conditions. We have a
clear pattern for mocking everything and anything - the mock module -
and we should be using this here. Start doing so, reworking a lot of the
tests in the process, in order to avoid messing with oslo.config and
triggering the issue entirely.
[1] https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2614
[2] https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2183
Change-Id: I468cef94185a1b59f379ca527050450e03664c67
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Closes-Bug: #1825435
This brings in a couple of new checks which must be addressed, many of
which involve a rather large amount of changes, so these are ignored for
now. A series of follow-up changes will resolved these.
'pycodestyle' is added as a dependency rather than it being pulled in
transitively. This is necessary since we're using it in tests.
Change-Id: I35c654bd39f343417e0a1124263ff31dcd0b05c9
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
The CheatingSerializer fixture used in nova tests keeps
the RequestContext.db_connection set on the context object
which otherwise wouldn't normally happen. The context object
can show up in error notification payloads because of how
nova.exception_wrapper.wrap_exception works. That payload
is eventually serialized for notifications and since the
cheated RequestContext.db_connection is set and cannot be
serialized, it results in a UserWarning from the
jsonutils.to_primitive method (called via JsonPayloadSerializer).
This will eventually result in failures when that UserWarning
is made into an error.
To fix this, we can pass a fallback method to to_primitive()
which will serialize a RequestContext object the same way that
RequestContextSerializer serializes a context - by simply
converting it to dict form.
Since this only affects test runs, because of using the
CheatingSerializer fixture, it should have no impact on
runtime serializations.
Error logging is added to the FakeNotifier since it's hard
to know what is wrong in the payload unless it is logged.
Also, the WarningsFixture is updated to make sure we don't
introduce new UserWarnings for the serialization issue.
The jsonutils.to_primitive() fallback method was added to
oslo.serialization via commit cdb2f60d26e3b65b6370f87b2e9864045651c117
in 2.21.1 so we have to bump our minimum required version
of that library as well.
Change-Id: Id9f960a0c7c8897dbb9edf53b4723154341412d6
Closes-Bug: #1799249
This change makes nova configure oslo.messaging's active call monitoring
feature if the operator increases the rpc_response_timeout configuration
option beyond the default of 60 seconds. If this happens, oslo.messaging will
heartbeat actively-running calls to indicate that they are still running,
avoiding a false timeout at the shorter interval, while still detecting
actual dead-service failures before the longer timeout value.
In addition, this adds a long_rpc_timeout configuration option that we
can use for known-to-run-long operations separately from the base
rpc_response_timeout value, and pre_live_migration() is changed to use
this, as it is known to suffer from early false timeouts.
Depends-On: Iecb7bef61b3b8145126ead1f74dbaadd7d97b407
Change-Id: Icb0bdc6d4ce4524341e70e737eafcb25f346d197
The only periodic task (_remove_stale_clients) in ClientRouter was
removed in change I10f374adca672576058c4dbab708c040d166df47 and this
isn't the correct way to run periodic tasks repeatedly anyway (this
will only run them once upon ClientRouter.__init__), so remove the
useless call.
This also removes a functional regression test which was testing only
this specific code path, where the intended periodic task ran only
once upon init of the compute service. All other periodic tasks in
compute are disabled by default in functional tests.
Related-Bug: #1627838
Change-Id: I3079c5ae4bd60de44f04e0136978a67d13e7a809
When the doc structure was changed the location of the notification
devref also changed. This patch updates the reference to this doc in
the AssertionError emited in the test if new legacy notification is
introduced.
Change-Id: Iff30752bac64801ad8950eea5861d2b230f30fdf
Based on the description of the notify_on_api_faults config option [1]
and the code that uses it [2] nova sends and api.fault notification
if the nova-api service encounters an unhandle exception.
There is a FaultWrapper class [3] added to the pipeline of the REST
request which catches every exception and calls the notification sending.
Based on some debugging in devstack this FaultWrapper never catches any
exception. Every REST API method is decorated with expected_errors
decorator [5] which as a last resort translate the unexpected exception
to HTTPInternalServerError. In the wsgi stack the actual REST api call is
guarded with ResourceExceptionHandler context manager [7] which translates
HTTPException to a Fault [8]. Then Fault is catched and translated to
the REST response [7]. This way the exception never propagates back to
the FaultWrapper and therefore the api.fault notification is never emitted.
Based on the git history of the expected_errors decorator this notification
was never emitted for v2.1 API and as the v2.0 API now supported with the
same codebase than v2.1 it is not emitted for v2.0 calls either. As nobody
reported a bug I assume that nobody tried to use this notification for a very
long time. Therefore instead of fixing this bug this patch propses to remove
the dead code.
See a bit more detailed description on the ML [9].
[1] 0aeaa2bce8/nova/conf/notifications.py (L49)
[2] 0aeaa2bce8/nova/notifications/base.py (L84)
[3] 0aeaa2bce8/nova/api/openstack/__init__.py (L78)
[5] 0aeaa2bce8/nova/api/openstack/extensions.py (L325)
[7] 0aeaa2bce8/nova/api/openstack/wsgi.py (L637)
[8] 0aeaa2bce8/nova/api/openstack/wsgi.py (L418)
[9] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118639.html
Change-Id: I608b6ebdc69d31eb2a11ac6479fa4f2e8c20f7d1
Closes-Bug: #1699115
The instance.live.migration.force.complete notificatons are missing from the
legacy notification whitelist as every test emiting them are mocked out
the rpc notifier call.
This patch adds the extra test coverage and whitelists the notifications
Change-Id: Idf20fb435a7d11ff93bcb137699a38f5e4562d4c
Closes-Bug: #1718485
The compute.metrics.update legacy unversioned notification was missing
from the whitelist because the only test for it was mocking out the
rpc notifier call, which is what validates the entries in the whitelist.
This adds the entry to the whitelist and fixes the test to actually
use the notifier code.
Change-Id: Ie278ca58c371b8e3bd4861e6c86ebcf30c6778ea
Closes-Bug: #1717943
The get_transport method was deprecated in the oslo.messaging
5.24.2 release in favor of get_rpc_transport and
get_notification_transport. We are already using get_notification_transport,
we just need to change to use get_rpc_transport.
Change-Id: Ie276fedef35f411cafbaf25c2148fea42ae01410
This removes the two deprecated by_instance() and by_host() methods
in ClientRouter and replaces them with calls to a single client()
routine. We stopped using the instance and host parameters in these
methods in an earlier patch to remove the caching in the router.
Related to blueprint cells-aware-api
Change-Id: Ifa7cd99a442ef0f8e9ad4a6b52982d0fedf1cd7e
Wasting time on generating the payload of the notifications is
unnecessary if the notification will not be emitted due to
configuration.
This patch depends on an oslo messaging improvement that adds
a way to check if a Notifier will emit the notification or not.
Depends-On: Ib992f5c20fef85224fb00823e1d8d9c6cff19bec
Depends-On: I979fb3385671aba6f4162ef991da8f0febe0068a
Change-Id: I3e6741d59df49e1e58409314008c2ed609fdedc1
Those are remnants from the oslo-incubator times. Also, oslo.messaging
deprecated [1] transport aliases since 5.2.0+ that is the minimal
version supported for stable/newton. The patch that bumped the minimal
version for Nova landed 3 months+ ago, so we can proceed ripping
those aliases from the code base.
[1] I314cefa5fb1803fa7e21e3e34300e5ced31bba89
Change-Id: Id8c0bc6733e930803b8e9fcf42a9a1f81dcbb356
Closes-Bug: #1424728
Some services (such as telemetry) actually consume the notifications. So
if one deploys a service that listens on the same queue as telemetry,
there will be race-conditions with these services and one will not get
the notifications that are expected at points.
To address this, one sets a different topic and consumes from there.
This is not possible with versioned notifications at the moment. And, as
services move to using that, the same need will arise.
So, this adds a configuration option to nova for enabling the
configuration of topics for this notifier.
Change-Id: I817ce4bae0dd37e0d06bd44f21ba81b3cb800548
Previously we had aimed to make things like compute RPC automatically
look up the InstanceMapping or HostMapping for the call being performed
to target appropriately. However, we cannot do that within the cell,
and even trying incurs some overhead. For now, just deprecate the
by_instance() and by_host() methods and honor what is in the context
(if set) and otherwise fall back to the default client. Make the context
target routines create and store the RPC transport and remove the caching
logic from the ClientRouter since we're removing its ability to do that.
Related to blueprint cells-aware-api
Change-Id: I10f374adca672576058c4dbab708c040d166df47
oslo.messaging allow dispatcher to restrict endpoint methods since
5.11.0 in d3a8f280ebd6fd12865fd20c4d772774e39aefa2, set with
DefaultRPCAccessPolicy to fix FutureWarning like:
"The access_policy argument is changing its default value to <class
'oslo_messaging.rpc.dispatcher.DefaultRPCAccessPolicy'> in version '?',
please update the code to explicitly set None as the value:
access_policy defaults to LegacyRPCAccessPolicy which exposes private
methods. Explicitly set access_policy to DefaultRPCAccessPolicy or
ExplicitRPCAccessPolicy.
"
Change-Id: I568d6bb8ec245059f09f853388c6fa3b30bded86
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
1) It checks that person who want to trace is trusted and knows
secret HMAC key.
2) It starts tracing in case of proper trace headers
and adds the first wsgi trace point with info about the HTTP request
* Add initialization of osprofiler on start of a service
Currently that includes oslo.messaging notifier instance creation
to send Ceilometer backend notifications.
oslo-spec: https://review.openstack.org/#/c/103825/
python-novaclient change: https://review.openstack.org/#/c/254699/
based on: https://review.openstack.org/#/c/105096/
Co-Authored-By: Boris Pavlovic <boris@pavlovic.me>
Co-Authored-By: Munoz, Obed N <obed.n.munoz@intel.com>
Co-Authored-By: Roman Podoliaka <rpodolyaka@mirantis.com>
Co-Authored-By: Tovin Seven <vinhnt@vn.fujitsu.com>
Implements: blueprint osprofiler-support-in-nova
Change-Id: I82d2badc8c1fcec27c3fce7c3c20e0f3b76414f1
Notifications related configuration options have been moved from the
default group to the ``notifications`` group.
Also, ``default_notification_level`` has been renamed to avoid naming
redundancy.
Blueprint centralize-config-options-ocata
Change-Id: I09dc358fabc84f7bf37d40d48b0652a10d9b8903
For testing cells, we will need to track the driver instances that
we give out by url. This normally just works with a conventional
oslo.messaging driver, but the fake driver keeps internal data
structures for simulating its bus. If we end up with clients creating
a new instance of the driver in the rpc switching code, we'll never
be able to send messages to services because we'll always have
private/separate data structures.
So, this makes the fixture wrap the transport creation stuff
and unify references by url. In order to make this work, some
retooling of rpc.init() is done, which makes it more in line with
the recent additions we had for wrapping transport initialization
per connection anyway.
For now, a lot of our tests can't handle the possibility of
multiple RPC connections due to them looking at the global
transport_url configuration. So for the moment, even though this
makes the fixture support multiple independent connections, we
collapse any such attempts down to a single connection to the
default broker.
Note: this requires a fix in oslo.messaging 5.14.0
Depends-On: I01b6f5a20ea9752da46a546a908bd38cf11da681
Change-Id: Icb63d7dabd17f3c5633387793f68a8ba20863a7e
In commit 4df0869, a ClientRouter was added to cache client connections
to cell message queues. The periodic task for removing stale clients
is called with an empty RequestContext, but this overwrites the copy
in thread local storage unless overwrite=False is specified.
This adds overwrite=False to the empty RequestContext to prevent it
from overwriting the thread local context. All of the other periodic
tasks use get_admin_context() which also creates a RequestContext with
overwrite=False.
Closes-Bug: #1627838
Change-Id: I48024952865fe017e0cb786567b5b445b05e7659
This moves the cell message queue switching code from the
RequestContext to the RPC layer where it's ultimately used.
Originally, it was thought that a separate DB query for
a CellMapping would occur per compute RPC API call in the
API cell and the context manager would be invoked to inject
the cell message queue transport to use for the RPC call.
Since compute RPC calls are based on the CellMapping of an
instance or a host, we could instead have generic functions
that take an instance or host and look up InstanceMapping or
HostMapping to get the CellMapping and return the
corresponding RPC client.
The RPC client objects are cached by CellMapping uuid and
expired clients are removed using a periodic task.
Co-Authored-By: Brian Elliott <bdelliott@gmail.com>
Depends-On: I6f211e9102f79418f9f94a15784f91c4150ab8a7
Change-Id: I96849888087f4b09433cb683a9eb4719d1c35c4c
A new versioned notification is emitted from the wrap_exception
decorator.
Change-Id: I8fcbce111537dae6ad3ba0961fb761540ae800ed
Partially-Implements: bp versioned-notification-transformation-newton
This adds message queue connection information to the RequestContext
which can be used by nova-api to communicate with a targeted cell
message queue with each query.
A function 'get_cell_client' can be called in rpc functions to enable them
to use message queue transport information from a RequestContext. The
function creates a rpc client object dynamically if message queue
connection information is found in the RequestContext and falls back on
the default rpc client.
Example usage:
def get_cell_client(self, context):
return rpc.get_cell_client(context, self.client)
def build_and_run_instances(self, ctxt, instance, host, image, ...)
cctxt = self.get_cell_client(ctxt).prepare(...)
cctxt.cast(...)
Implements blueprint cells-mq-connection-switching
Change-Id: Idef670d5b73c9cef8501a0593eccd785b708bd2b
According the repeated warning during py34 test runs 'topic' has been
deprecated in favor of 'topics' for oslo.messaging Notifiers. This
updates the Nova usage in order to not get yelled at.
Depends-On: Iedd1f5264e87e506bd3a1c09614aa1dc891dc685
Change-Id: Id028e6a406fa871e6561f36f95ed748afc320a20
The single option in nova.rpc is moved to the central place.
As this module contains a single option this patch also enhances
the help text of it.
Implements: bp centralize-config-options-newton
Closes-bug: #1565753
Change-Id: Ib6030c67b315a3b8d0d4854b4b6f1b969be6c00c
As we agreed on the Mitaka midcycle no new legacy notification
event_type are allowed in Nova and every new notification shall
use the versioned notification framework.
This patch adds a validation step into the legacy notification
sending call chain. It checks the event_type against a whitelist.
During unit and functional testing an exception is raised if the
event_type is not known and therefore forbiden. This check is not
100% foolproof as it only detects the new event_type if there is
a test that triggers the sending of it and this check is not capable
of detecting such event_types if the test directly mocks
nova.rpc.get_notifier.
During normal operation a WARNING is logged so that a logstash query
can be set up to detect the rest of the cases.
Change-Id: Ia6600868aa7b3a22cb8c020503f49dce6a4c2c2b
In oslo.messaging 2.9.0, the RPC and notification buses were decoupled
into separate transports. This changes over the Nova notifier to use the
notification transport instead of the original transport (which is now
used for RPC).
Change-Id: I595d9dd7986a87dfc93e3579d4498b2d9838a4d8
Partially-Implements: bp oslo-for-mitaka
A new config option is introduced called 'notification_format'
which specifies the format of the notifications emitted, possible
values are 'versioned', 'unversioned', 'both'. The default value is 'both'.
Also to help consumers the versioned notifications will be emited to
a separate topic called 'versioned_notifications'.
DocImpact: new config option notification_format
Partially-Implements: bp versioned-notification-api
Change-Id: Ie45c03175800bb14269f3976b03a53488e084339
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware
Change-Id: I3e2eb147b321ce3e928817b62abcb7d023c5f13f
We need to pass the service parameter else the publisher_id is
set to "None.node-1". There are many code paths that end up
in get_noftifier, so let's just handle it there just like we
default to CONF.host. Also the self.notifier in the
resource_tracker was not being used, so get rid of it.
Closes-Bug: #1416435
Change-Id: I8838b961e49496cc85913a04bd99f9845ff308c3
Very simple import change in each file, just touches a
whole lot of files. jsonutils has graduated into a stand alone
library and has been removed from the oslo-incubator repository.
We should be using the library for all projects.
Change-Id: I09d780a47d377871f357654603ee4b0fa0169d66
When a notification payload is passed to the oslo.messaging the
payload must be correctly serialized.
This change encode the payload into json as the old rpc did.
Closes bug: #1279881
Change-Id: I99fc2a4b0e0146c9faa6abeddbd15cb24169e024
RequestContext should be serialized when sent via oslo.messaging. The
serializer is correctly used for the general RPC mechanism, but has been
forgotten in the notifier. This patch fixes that.
Change-Id: I56fa8022e34c0e80835e3bde940fda99ed0f9ba8
Closes-Bug: #1275771
We don't need to have the vi modelines in each source file,
it can be set in a user's vimrc if required.
Also a check is added to hacking to detect if they are re-added.
Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6
Closes-Bug: #1229324
The oslo.messaging library takes the existing RPC code from oslo and
wraps it in a sane API with well defined semantics around which we can
make a commitment to retain compatibility in future.
The patch is large, but the changes can be summarized as:
* oslo.messaging>=1.3.0a4 is required; a proper 1.3.0 release will be
pushed before the icehouse release candidates.
* The new rpc module has init() and cleanup() methods which manage the
global oslo.messaging transport state. The TRANSPORT and NOTIFIER
globals are conceptually similar to the current RPCIMPL global,
except we're free to create and use alternate Transport objects
in e.g. the cells code.
* The rpc.get_{client,server,notifier}() methods are just helpers
which wrap the global messaging state, specifiy serializers and
specify the use of the eventlet executor.
* In oslo.messaging, a request context is expected to be a dict so
we add a RequestContextSerializer which can serialize to and from
dicts using RequestContext.{to,from}_dict()
* The allowed_rpc_exception_modules configuration option is replaced
by an allowed_remote_exmods get_transport() parameter. This is not
something that users ever need to configure, but it is something
each project using oslo.messaging needs to be able to customize.
* The nova.rpcclient module is removed; it was only a helper class
to allow us split a lot of the more tedious changes out of this
patch.
* Finalizing the port from RpcProxy to RPCClient is straightforward.
We put the default topic, version and namespace into a Target and
contstruct the client using that.
* Porting endpoint classes (like ComputeManager) just involves setting
a target attribute on the class.
* The @client_exceptions() decorator has been renamed to
@expected_exceptions since it's used on the server side to designate
exceptions we expect the decorated method to raise.
* We maintain a global NOTIFIER object and create specializations of
it with specific publisher IDs in order to avoid notification driver
loading overhead.
* rpc.py contains transport aliases for backwards compatibility
purposes. setup.cfg also contains notification driver aliases for
backwards compat.
* The messaging options are moved about in nova.conf.sample because
the options are advertised via a oslo.config.opts entry point and
picked up by the generator.
* We use messaging.ConfFixture in tests to override oslo.messaging
config options, rather than making assumptions about the options
registered by the library.
The porting of cells code is particularly tricky:
* messaging.TransportURL parse() and str() replaces the
[un]parse_transport_url() methods. Note the complication that an
oslo.messaging transport URL can actually have multiple hosts in
order to support message broker clustering. Also the complication
of transport aliases in rpc.get_transport_url().
* proxy_rpc_to_manager() is fairly nasty. Right now, we're proxying
the on-the-wire message format over this call, but you can't supply
such messages to oslo.messaging's cast()/call() methods. Rather than
change the inter-cell RPC API to suit oslo.messaging, we instead
just unpack the topic, server, method and args from the message on
the remote side.
cells_api.RPCClientCellsProxy is a mock RPCClient implementation
which allows us to wrap up a RPC in the message format currently
used for inter-cell RPCs.
* Similarly, proxy_rpc_to_manager uses the on-the-wire format for
exception serialization, but this format is an implementation detail
of oslo.messaging's transport drivers. So, we need to duplicate the
exception serialization code in cells.messaging. We may find a way
to reconcile this in future - for example a ExceptionSerializer
class might work, but with the current format it might be difficult
for the deserializer to generically detect a serialized exception.
* CellsRPCDriver.start_servers() and InterCellRPCAPI._get_client()
need close review, but they're pretty straightforward ports of code
to listen on some specialized topics and connect to a remote cell
using its transport URL.
blueprint: oslo-messaging
Change-Id: Ib613e6300f2c215be90f924afbd223a3da053a69
in the bugfix for lp804063. That bugfix should have only added auto_delete = True to FanoutPublisher to match the previous change to FanoutConsumer.
TopicConsumer isn't exclusive or auto_delete, so TopicPublisher has to still match.
Tobias Urdin