Blueprint remove-namespace-packages
Depends-on: I2eeef93ee2e61a721c69f62add819f93f62f077d
for openstack/ceilometer
Depends-on: I26390dd908769be5f1a5b70be22d3b98e3a45563
for openstack/ceilometermiddleware
Depends-on: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
for openstack/glance
Depends-on: I029c3260051aa48cfaae428c096c1ac7b43b2bd2
for openstack/ceilometermiddleware
Change-Id: I8c5595bbafa82db33f62fa47d214f5cb756a2639
The context manager in the executor fit only for the blocking executor.
Even the dispatcher needs to run code before and after the application
callback, eventlet and future executors have to run the pre/post code
into the main thread and can run the callback into an other thread,
and that force them to run __enter__ and __exit__ manually and deal
the exception path.
This change adds a helper object instead of the context manager.
It is designed to be explicit on what must be executed
before and after the callback and what can be done in a thread or not.
All the executor code is now in the impl_pooledexecutor.py
and use the futures "PoolExecutor" API.
This use futurist to provide a eventlet and aioeventlet futures friendly
object.
Change-Id: I8cd7640f36beeda47560e3c82671bad3530e38d1
Raise an exception if a caller passes a something for the
version that cannot be parsed as a version.
Change-Id: I167849b1bc6c4d946de42c6459b3577a361e9606
Related-Bug: #1467087
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: I05a408b76d4f31b803769a27759e91df770511bb
Now that python-qpid-proton has taken an approach similar to zmq's, it's
possible to install it in boxes where the C libraries are not present in
the system. This patch takes advantage of that to enable amqp protocol's
tests everywhere.
Change-Id: Ibce4c791aafadf45fa876d6a4f5373fde94769ff
oslotest 1.7.0 breakes oslo.messaging unittests, because it doesn't add
mox to six.moves anymore - see change Ic59c73abb9b09cb594bf7df4173d7f99f81d526c
This patch fixes imports on test run.
Change-Id: Ia40b7733bd2e74b1a1703f2e1a5245f01debbd36
This change causes the message sender to block until the messaging
infrastructure (e.g. the broker) assumes ownership of the message (or
fails to accept it). If the message is accepted, then the sender will
either pend for a response (in the case of RPC), or simply return (in
the case of notification). If the message is rejected by the
messaging infrastructure a MessagingException will be raised at the
sender.
Change-Id: I3f4a1ed1c17e18f6d629f16e6b5c99de45b083d6
Closes-Bug: #1377228
When a consumer is declared after we have started to consume
amqp, its queue is never consumed.
This fixes that.
Closes bug: #1450342
Change-Id: I9f2e7d83283504dfe762ac88384efde0f7b52d47
The consumer loop is over engineered, it returns unused return,
iterconsume creates an iterator directly consumed by 'consume' without
special handling, and in some case kombu error callback are called when
the iterator is stopped and log useless error.
And in reality the consumer is always called when limit=1.
This change simplifies that, by removing the loop and removes all
returns stuffs.
Closes bug: #1450336
Change-Id: Ia2cb52c8577b29e74d4d2b0ed0b535102f2d55c7
To avoid creating a new ZMQ connection for every message sent
to a remote broker, implement pooling and re-use of ZmqClient
objects and associated ZMQ context.
A pool is created for each remote endpoint (keyed by address);
the size of each pool is configured using rpc_conn_pool_size.
All outbound message client connections are pooled.
Closes-Bug: 1384113
Change-Id: Ia55d5c310a56e51df5e2f5d39e561a4da3fe4d83
The mock module has been added to Python 3.3 as unittest.mock.
The third party mock module doesn't seem to be maintained anymore: the
last commit was in April 2013.
unittest.mock is more recent, has less bugs and is compatible with
Python 3.4. There are bugs on Python 3 in the third party mock module,
examples:
* https://code.google.com/p/mock/issues/detail?id=225
* https://code.google.com/p/mock/issues/detail?id=234
Oslo Messaging hits these issues when running tests on Python 3.4.
Import oslotest to setup six.moves for mock.
Change-Id: Ic160101695cea67eb9bbbfcaddb8d3dac64e6804
When publish_errors is True, and using `messaging` notify driver,
produces a infinite loop that report sending notification error.
The reason is we always passing None to the content argument in
logger handlers (LoggingNotificationHandler, LoggingErrorNotificationHandler),
when pack the context object which expected as a dict, raise TypeError
exception, so result in infinite retry looping.
To match what document said, passing a empty dict rather than None. Also
modify unittests to make sure context is a empty dict.
Change-Id: Ic2c4c43f5bbafa8107ea370ba959da16cfa4a24c
Closes-bug: #1346466
Url constructed for kombu cannot contain ipv6 address without
brackets. The url.hostname returns hosts without brackets so
we have to readd them.
Change-Id: Ifb1d358a67655af99a84b77ca813fc2dd87d87dc
For all transport errors, we first got a log of a failure with a
backtrace like 'Fail to publish message on topic' or 'fail to consume
messages' and then another message like 'AMQP is unreachable'.
But in most case, we retry to consume/publish messages when the reason
is a connection lost.
So, now we don't log the failure message anymore in case of connection lost,
but just message related to rabbit deconnection/reconnection.
The error message and the backtrace are only logged in case of we really fail to
publish a message.
Change-Id: Ifa1b04b348b347bad0b2abec3e759a2ce7815b86
AMQP offers a heartbeat feature to ensure that the application layer
promptly finds out about disrupted connections (and also completely
unresponsive peers). If the client requests heartbeats on connection, rabbit
server will regularly send messages to each connections with the expectation of
a response.
To acheive this, each driver connection object spawn a thread that
send/retrieve heartbeat packets exchanged between the server and the
client.
To protect the concurrency access to the kombu connection between the
driver and this thread use a lock that always prioritize the
heartbeat thread. So when the heartbeat thread wakes up it will acquire the
lock quickly, to ensure we have no heartbeat starvation when the driver
sends a lot of messages.
Also when we are polling the broker, the lock can be held for a long
time by the 'consume' method, so this one does the heartbeat stuffs itself.
DocImpact: 2 new configuration options for Rabbit driver
Co-Authored-By: Oleksii Zamiatin <ozamiatin@mirantis.com>
Co-Authored-By: Ilya Pekelny <ipekelny@mirantis.com>
Related-Bug: #1371723
Closes-Bug: #856764
Change-Id: I1d3a635f3853bc13ffc14034468f1ac6262c11a3
Using the import of impl_zmq to decide whether to skip tests is
unsafe as if something breaks in the import, then the tests are
skipped, potentially hiding a bug.
Switch to using the import of the zmq package itself to decide
whether tests should be executed or not; this should cover the
currently unsupported Python3 test coverage.
Change-Id: I7d65506845b039f5e2be4c460bede977122deca9
All drivers options are current stored into the DEFAULT group.
This change makes the configuration clearer by putting driver options
into a group named oslo_messaging_<driver>.
Closes-bug: #1417040
Change-Id: I96a9682afe7eb0caf1fbf47bbb0291833aec245b
If rabbit dies, the consumer can be disconnected before the publisher
sends, and if the consumer hasn't declared the queue, the publisher's
will send a message to an exchange that's not bound to a queue, and
the message wll be lost. Setting passive=True will cause the
publisher to fail and retry if the consumer hasn't declared the
receiving queue yet.
Co-Authored-By: Noel Burton-Krahn <noel@burton-krahn.com>
Closes-Bug: #1338732
Change-Id: I5ba4d311b97236d3a85a9f5badff61f12b08c12d
This removes the functionnal tests from the legacy
directory and run the ones in the new directory with tox.
Change-Id: Ibbfd6c946e25435ff37e459bf5e82565a1e21778
When a url looks like: kombu+qpid:///host:port/ the driver fail
to load.
This change fixes that, adds a warning message that this kind of
URL is experimental and not yet supported.
Also our Consumer code use internal kombu API that can be optionnal
implemented by the kombu transport (message_to_python),
so check that this one exists before using it.
In the future, we should use kombu.messaging.Consumer/Publisher
instead of Consumer/Publisher implementation to avoid such hack...
Change-Id: I066d57c23bff922c5734ab036b6ca8e1608e5c6a
A number of the ZMQ test cases shared the same setUp method; refactor
and introduce a base test case class to share this code across all
appropriate ZMQ test cases.
Change-Id: I59300464af001c343efcd4f3f33d34c972da2b87
Fix test_notifier: import notifier from oslo.messaging.notify, not from
oslo_messaging.notify.
Remove tests/drivers/test_pool.py and tests/notify/test_notifier.py
because they only import oslo_messaging.
Change-Id: I6289dd0c48b709c71c8befb08e275675dc8d4286
The Notification Listener Tracker class is not threadsafe,
so when a test stop an already stopped listener this one
can be restarted, due to concurrency access of the threads list and
concurrency execution of the start/stop/wait method of the notification
listener.
This result of a lockup of the test or a listener can continue to
got unepxected message.
This change fixes that by never stop the tread with the tracker callback
but only manually with the test.
This test also rename some 'Listener' to 'Server', to not mismatch
the driver listener from the notification listener.
Closes-bug: #1410902
Change-Id: I4777c7dd0ba71c61850d36641e85f33f9461e9c1
In ZmqDriver's listen method, it calls create_consumer
three times. After the first call, the keys related to this
topic in redis reads like this:
"topic": set(["topic.host"])
"topic.host": ""
If the second call, it tries to add the following keys:
"topic.host": set(["topic.host.host"])
"topic.host.host": ""
But the key "topic.host" already exists as a string type.
So the error occurs.
To resolve this problem, change the value of keys from string
to set.
Change-Id: Ic801393d492d2656fcfd8b87f1d2efc6ab3bbd62
Closes-Bug: #1290772
Tests now ensure that eventlet monkey patching is enabled before
importing the qpid module, otherwise qpid will hang.
Currently, qpid randomly hangs, depending on the order of the Python
module import, which probably depends on the local file modification
time.
Closes-Bug: #1409899
Change-Id: I516c580faef04f55fcc095d22adbfabcfa001da7
Move the public API out of oslo.messaging to oslo_messaging. Retain
the ability to import from the old namespace package for backwards
compatibility for this release cycle.
bp/drop-namespace-packages
Co-authored-by: Mehdi Abaakouk <mehdi.abaakouk@enovance.com>
Change-Id: Ia562010c152a214f1c0fed767c82022c7c2c52e7
When the duration of the timeout timer used in the rabbit driver.is
None and we want that the timer return a maximum of N secs it return None
(infinite) instead of N.
This change fixes that.
Closes-bug: #1408370
Change-Id: I7f4cb3075f776c63aa7dc497173677f92b68c16d
The concurrent.futures module is one of the ways
that async activities can be done in the future,
so we should try to work on getting to that future
by using more futures. To enable this (as well as
to enable getting off eventlet), add a thread pool
based executor which will process incoming messages
using the pool.
Also begins adding according docs as well for the
different types of executors that are available.
Change-Id: I1482fd70abbf69f4e2994597c5e95d91fecb815e
This change improves test coverage of the rabbit driver for new
value allowed in rpc_backend config option.
Change-Id: I52a8a8beb5b0d32647f52bb21abd5f7164dac2df
Basic functional and unit tests for zmq driver.
Note as the zmq driver is directly dependent on eventlet, this
change also updates the notify logger tests to remove the
direct dependency on threading which was being monkey patched,
causing test failures.
As the zmq driver has a direct dependency on eventlet, tests are
skipped under py3.
Co-Authored-By: Kapil Thangavelu <kapil.thangavelu@canonical.com>
Co-Authored-By: Edward Hope-Morley <edward.hope-morley@canonical.com>
Change-Id: I93b8b2e92d0f2a353d3357a5e61f6d472ec84944
Partial-bug: #1302941
It's up to the driver to set a suitable timeout for polling the broker,
this one can be different that the one requested by the driver
caller as long as the caller timeout is respected.
This change also adds a new driver listener API, to be able
to stop it cleanly, specially in case of timeout=None.
Closes bug: #1400268
Closes bug: #1399257
Change-Id: I674c0def1efb420c293897d49683593a0b10e291
This change reintroduces the fake_rabbit only for backward compatibility,
but mark it as deprecated.
Now, to use the kombu in-memory driver (that is not thread safe) we must
set the transport_url config option to 'kombu+memory:////" or the
rpc_backend to kombu+memory.
Or we can use the fake driver of oslo.messaging by setting the
transport_url to 'fake:///' or the rpc_backend to 'fake'
This is effectively reverting commit bcb3b23b8f.
Closes-bug: #1399085
Change-Id: I7b6fb3811fc6f695f75ecd350e04e69afd26c428
Doug Hellmann