We now expect context objects to support returning a redacted copy of
themselves.
As a related cleanup, removed the practice entirely of using
dictionaries to represent contexts in unit tests and the logging driver.
As part of developing this change, I discovered code in Glance (and
potentially other services) which explicitly pass {} in lieu of a
context when notifying; so we now properly handle dictionaries as
contexts.
To ensure we have the method required; require oslo.context 5.3.0 or
newer.
Change-Id: I894f38cc83c98d3e8d48b59864c0c7c2d27e7dcd
We encountered bug 2037312 in unit tests when attempting to get this
change rolled out. Heat apparently will attempt to set is_admin using
policy logic if it's not passed in for a new context; this breaks as the
context we are requested doesn't have all the needed information to
exercise the policy logic.
is_admin is just a bool; it's not sensitive; easiest route forward is to
add it to the safe list
Closes-bug: 2037312
Change-Id: I78b08edfcb8115cddd7de9c6c788c0a57c8218a8
Publishing a fully hydrated context object in a notification would give
someone with access to that notification the ability to impersonate the
original actor through inclusion of sensitive fields.
Now, instead, we pare down the context object to the bare minimum before
passing it for serialization in notification workflows.
Related-bug: 2030976
Change-Id: Ic94323658c89df1c1ff32f511ca23502317d0f00
The rabbit backend now applies the [oslo_messaging_notifications]retry,
[oslo_messaging_rabbit]rabbit_retry_interval, rabbit_retry_backoff and
rabbit_interval_max configuration parameters when tries to establish the
connection to the message bus during notification sending.
This patch also clarifies the differences between the behavior
of the kafka and the rabbit drivers in this regard.
Closes-Bug: #1917645
Change-Id: Id4ccafc95314c86ae918336e42cca64a6acd4d94
Oslo.utils's fnmatch module was added to fix the py2.7 fnmatch module
who was not thread safe [1]. Python 2.7 is no longer supported so now we
can use the stdlib's fnmatch module and deprecate the one of oslo.utils.
[1] https://bugs.python.org/issue23191$
Change-Id: Id5381a0a5216783f0df594b126786947db16a8d1
Remove six, the python 2/3 compatibility library. It's not needed
anymore since the repo is python3 only.
Remove a now unneeded hacking test.
Change-Id: I40522c4accb4aaf8115d11fee8b081e2d991cb4d
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.
Update local hacking check for new flake8 version.
Blacklist:
W504 line break after binary operator
Fix:
E741 ambiguous variable name
E117 over-indented
E305 expected 2 blank lines after class or function definition, found 1
F841 local variable 'e' is assigned to but never used
W605 invalid escape sequence '\.'
Change-Id: I99d574ca6569f1f177d2c5ce1011f269f4343619
The blocking executor has been deprecated in Pike and marked for removal
in Rocky, but some user like Mistral asked us to wait before. We decided
to remove this executor for Train or next cycle, now we are in the
Ussuri and after some researchs on usage I think we can go ahead.
This patch drop the deprecation warnings, related unit tests and
set the server with the threading executor is the default executor.
Change-Id: If07bab61ee2b148658b88be98b12f8539f274efe
Closes-Bug: #1715141
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.
Change-Id: I9ddb6595fc52e46ed8844e39d2fa71029c90b65c
Closes-Bug: #1674567
To facilitate removing rpc_backend support, disentangle the aliases
code by removing it to simplify the code and the tests. Aliases were
first depreacted in 5.20.0 during Pike.
Change-Id: I7d80cd050ce1eca2c6b6c38f0fac29d090b90bf3
Closes-Bug: #1424728
there's only one scenario where the generator actually needs to be
forced into a list. don't waste time doing something unnecessary.
Change-Id: Iec897b1f6fed3d88b798eb673779e7c68ff930ec
groupby requires items to be sorted as it's clumping consecutive
items together. also, just use itemgetter instead of lambda
Change-Id: I1676fc60ebc2c9ff03ea00cc67061fdfb3b50868
Since RPC and notifications can have different backends, it is
useful to warn users if they use a notification transport in RPC
and vice versa. This patch introduces RPCTransport and
NotificationTransport subclasses of Transport, so it's easier to
add different behavior for them if need be.
Related-Bug: #1680192
Change-Id: Iab60544d69053c8e74c28a2d5c84665be749013f
Using blocking executor is not recommended for application. But it looks
like some use it just because it's the default and are not aware their
should change it despite of documentation and logging message.
Choosing the application threading model is an important step
of an application.
This change deprecates it, in the future we will just make executor
mandatory. This will ensure that application make a choice.
Also this will reduce headache of oslo.messaging developers to
make the driver code working in a sync and async.
And to finish test coverage of blocking executor is 0%...
This rework some tests to remove logging.captureWarnings() that can
catch unwanted warning of other tests. Tests mocks warning instead.
Related-bug: #694728
Change-Id: Ic67164d12e7a9bed76d6e64ca2ced12e3984ff5f
Be specific regarding which characters can be used in the 'username'
and 'password' fields of the URL.
Change-Id: I9a2a801b3dd86fceffb05f4d52c715748c88d0cb
Closes-Bug: #1693967
The get_rpc_transport wraps get_transport to unify the API in
anticipation of comprehensive separation of RPC and Notification
messaging backends.
Related-Bug: 1680192
Change-Id: Ic6af07b98ff43806c2af38a3ba129991f1e0ec86
When the underlying RPC is configured with rabbit along
with oslo_messaging_notifications, currently the retry is
not supported. There are retry support specific to drivers,
it's better to have the support uniform across the drivers,
by adding configuration retry on the oslo_messaging_notifier.
Closes-bug: 1666364
Change-Id: I8d547ef2a9e8b086f94a1efca17735f3f3cdaac1
oslo.messaging supports three types of executors: blocking, eventlet
and threading, so it's good to list all of them in docstring.
Change-Id: Ief14d54b5f029cf3e9c79334e405dca5cf2dc13d
15cd99050c added this alias
to workaround an issue with neutron using the internal
path in liberty. That was fixed in neutron in mitaka:
225663888e71d44e9732951f515da925b8799b74
The alias was left for liberty, but liberty is end of life
now so we can remove this.
Change-Id: I145921b98d48e03df38ebcbaebc1a4375911f9ff
Generating the payload of a notificatiton might be expensive due
to database access. This expensive step is unnecessary if the
notification driver is set to noop because it means that the
generated payload will not be sent.
This patch makes it possible for the user of the notifier to check
if the notifier will emit any notification before the payload
is generated.
Cinder already implemented similar behavior in
I77f655d3ef90088ce71304da5d4ea7b543991e90 and nova also plans
to do it in I3e6741d59df49e1e58409314008c2ed609fdedc1
Change-Id: Ib992f5c20fef85224fb00823e1d8d9c6cff19bec
1.As mentioned in [1], we should avoid using six.iteritems to achieve iterators.
We can use dict.items instead, as it will return iterators in PY3 as well.
And dict.items/keys will more readable. 2.In py2, the performance about
list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
Change-Id: Ia235afc3532f62f265f91ca46d2306c72fc2a2a2
'Notification Listener' doesn't work.
I fixed the example which is described in developer guide to work fine.
These are the main points of this correction.
* specify namespace for NotificationFilter.
* set 2 blank lines between class definitions according to E302 of pep8.
* change argument of 'pool' to keyward arguemnt, because 4th argument of
'get_notification_listener' requires the name of message executor.
Change-Id: I75800309f2294fe9f17ea2a2b1638de57ef2922e
Closes-Bug: #1618394
Rework the inline documentation for executors, RPC servers and
clients, notifiers and notification listeners for clarity and flow.
Change-Id: If4f1db853a7fc85340177fd2c9c43a479d72459d
When data[k] is None during match check, we are getting annoying
TypeError exceptions like: "TypeError: expected string or buffer".
This patch adds a check that data[k] is something the regex can do
matching.
Change-Id: I75c8f602a59a90aeb5a6973f2e9e25708194ad78
This was previously undocumented. Most of the information was taken from the
docstring for get_transport
Change-Id: I92652f4de9dd1e59ae5c3f9ce39e5ee808f24c7d
1) try to catch all possible exceptions during message acknowledging
and dispatching to send ack/nack when we can to do it
2) improve logging in case of exceptions during message acknowledging
and dispatching
Depends-On: I2d230d49e5aff6ab4d84ab8c3d2834f85e3405eb
Change-Id: I41a768c5624fa2212257ce20bf9a67d09de0c4ab
fnmatch is not thread safe in python <2.7.10, let's use the safe
one in oslo.utils
Change-Id: If3d480436506ecc5f3c2fcbd51ceaac59f2b7dec
ref: https://hg.python.org/cpython/rev/fe12c34c39eb
1) Add MessageHandler base interface for on_incoming_callback replacement
2) Move message_handler parameter form Listener's __init__() to start()
3) Remove wait method from listener
Change-Id: Id414446817e3d2ff67b815074d042a9ce637ec24
The transport_url may contain credentials to the message queue; For
this reason, this option should be secret for it to not leak into the
logs.
Closes-Bug: #1567233
Change-Id: I49c641a2662976d7220e4222e3c4a4d2586b1336
Current Listener interface has poll() method which return messages
To use it we need have poller thread which is located in MessageHandlerServer
But my investigations of existing driver's code shows that some implemetations have
its own thread inside for processing connection event loop. This event loop received
messages and store in queue object. And then our poller's thread reads this queue
This situation can be improved. we can remove poller's thread, remove queue object
and just call on_message server's callback from connection eventloop thread
This path provide posibility to do this for one of drivers and leave as is other drivers
Change-Id: I3e3d4369d8fdadcecf079d10af58b1e4f5616047
Dispatcher should be responsible for routing message to the
callback method of endpoint object and returning result back to the
server only. But now it is also responsible for sending reply,
ack/reque messages etc.
Also this patch makes small improvements:
1) Notification dispatcher now requeue message if endpoint raises exception
2) unstable behaviour of test_mask_passwords test is fixed
Change-Id: I5f23e23644e90919cb67f81fc306ee85c5e09974
This reverts commit bd81d09c02.
I understand that the change was supposed to fix something, but instead it broke all tests on Python 3!?
It's wrong to replace blindly json.dumps() with jsonutils.dump_as_bytes(). In oslo messaging, the result is usually used as a value in a dictionary, and then the whole dictionary is passed to a second serializer which also serialize to JSON.
Sorry, I don't understand everything, but at least I see that tests passed on py3 before the change, and started to fail with the change.
Maybe json(utils).dumps() is misused in some places, but in this case, you should write a change which only fix these specific places, not replace all calls to dumps().
Change-Id: Icd54ee8e3f5c976dfd50b4b62c7f51288649e112
Looks like there is a disconnect between the __init__
parameter 'topic' in Notifier and what we need when
we look up a driver. We should allow multiple topics
to be specified as a new topics parameter and pass
that along directly.
Change-Id: Id89957411aa219cff92fafec2f448c81cb57b3ca
Since the json.dumps() returns Unicode string On Python 3, to
ensure that the result type is bytes on Python 2 and Python 3
that if the result is used for the message body, let's replace
json.dumps() / oslo_serialization.jsonutils.dumps()
with
oslo_serialization.jsoutils.dump_as_bytes()
Change-Id: I0e0f6b715ffc4a9ad82be52e55696d032b6d0976
The mask_dict_password method is added from oslo_utils 3.4.0, and it's
about five times faster than the mask_password method, so it's better
to use mask_dict_password instead of mask_password.
Change-Id: I6629ed1232f2e6f1c72b9f13b361e2f9e14b50a4
Jay Faulkner