Commit Graph

268 Commits

Author SHA1 Message Date
Doug Hellmann 03265410e0 Remove oslo namespace package
Blueprint remove-namespace-packages

Depends-on: I2eeef93ee2e61a721c69f62add819f93f62f077d
for openstack/ceilometer
Depends-on: I26390dd908769be5f1a5b70be22d3b98e3a45563
for openstack/ceilometermiddleware
Depends-on: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
for openstack/glance
Depends-on: I029c3260051aa48cfaae428c096c1ac7b43b2bd2
for openstack/ceilometermiddleware

Change-Id: I8c5595bbafa82db33f62fa47d214f5cb756a2639
2015-07-16 17:47:16 +00:00
Doug Hellmann a6d068a984 Add oslo.messaging._drivers.common for heat tests
Some of the heat unit tests import code from a private part of
oslo.messaging. With the namespace package change, that private part was
no longer being exposed. This change introduces the needed module so we
can release oslo.messaging with the namespace changes without breaking
heat, to give the heat team time to update their tests.

Partial-Bug: #1410196
Change-Id: Ib3ea106646f75b17665877b08ccbd382c5698714
2015-01-13 08:56:36 -05:00
Doug Hellmann e55a83e832 Move files out of the namespace package
Move the public API out of oslo.messaging to oslo_messaging. Retain
the ability to import from the old namespace package for backwards
compatibility for this release cycle.

bp/drop-namespace-packages

Co-authored-by: Mehdi Abaakouk <mehdi.abaakouk@enovance.com>
Change-Id: Ia562010c152a214f1c0fed767c82022c7c2c52e7
2015-01-12 12:50:41 -05:00
Jenkins 8102f25dd2 Merge "Fix some comments in a backporting review session" 2015-01-08 19:23:39 +00:00
Jenkins c45ad9ebfe Merge "Add a info log when a reconnection occurs" 2015-01-08 19:22:13 +00:00
Mehdi Abaakouk 31a149a4df Add a info log when a reconnection occurs
This change adds a info log when a reconnection occurs
like we have before.

Change-Id: I3b7d86894efac09e27125c368fc83e83c36b630e
2015-01-07 21:30:22 +00:00
Mehdi Abaakouk 44132d4344 rabbit: fix timeout timer when duration is None
When the duration of the timeout timer used in the rabbit driver.is
None and we want that the timer return a maximum of N secs it return None
(infinite) instead of N.

This change fixes that.

Closes-bug: #1408370

Change-Id: I7f4cb3075f776c63aa7dc497173677f92b68c16d
2015-01-07 21:30:01 +00:00
Mehdi Abaakouk c18f9f7c61 Don't log each received messages
oslo.utils.strutils.mask_password take too much time on
big payload (nova-conductor can receive payload ~ 66k in
largeops jobs for example).

This change removes this logging until we make mask_password
more efficient or we have a smarted the oslo.messaging logging to
not log everything.

Change-Id: Ib1f1d70c5cb820e8ff2de10e6064037808ea1f3a
Closes-bug: #1408362
2015-01-07 18:27:32 +01:00
Jie Li 3e2d142a87 Fix some comments in a backporting review session
* add i18n support to some exception messages
  * remove the return value of DecayingTimer.start()

Relates to review: I898a236a384b3466147026abc7a1ee21801e8ca1

Change-Id: I7adf5478732f1f46db1009b059b66ff8af6ecdc3
2014-12-27 12:01:25 +08:00
Jenkins 84f18e8167 Merge "safe_log Sanitize Passwords in List of Dicts" 2014-12-23 23:27:05 +00:00
Jenkins c575df2fdb Merge "Enable IPv6-support in libzmq by default" 2014-12-21 03:48:36 +00:00
Jenkins 39e4a0bcc4 Merge "Add a thread + futures executor based executor" 2014-12-21 03:44:32 +00:00
Jenkins 4b1fcd3622 Merge "Fix incorrect attribute name in matchmaker_redis" 2014-12-19 09:53:43 +00:00
Frode Nordahl c40ba040a6 Enable IPv6-support in libzmq by default
Change-Id: I53790e5492ebf026e0b331ecb3c294da89603540
Closes-Bug: 1389137
2014-12-19 09:41:46 +01:00
Joshua Harlow 372bc4947e Add a thread + futures executor based executor
The concurrent.futures module is one of the ways
that async activities can be done in the future,
so we should try to work on getting to that future
by using more futures. To enable this (as well as
to enable getting off eventlet), add a thread pool
based executor which will process incoming messages
using the pool.

Also begins adding according docs as well for the
different types of executors that are available.

Change-Id: I1482fd70abbf69f4e2994597c5e95d91fecb815e
2014-12-18 23:49:53 -08:00
Mehdi Abaakouk 56a9c55a3f safe_log Sanitize Passwords in List of Dicts
Sanitizes password fields found in lists of dicts for messages
before logging.

This change uses oslo.utils.strutils.mask_password to do it.

Change-Id: I7cd1e53e2ced7ebf9c5942b7a0dbbeb991acab4d
Closes-Bug: #1268459
2014-12-16 16:17:46 +00:00
Jenkins fa68eaa280 Merge "Add functional and unit 0mq driver tests" 2014-12-11 09:55:31 +00:00
James Page cd71c47d32 Add functional and unit 0mq driver tests
Basic functional and unit tests for zmq driver.

Note as the zmq driver is directly dependent on eventlet, this
change also updates the notify logger tests to remove the
direct dependency on threading which was being monkey patched,
causing test failures.

As the zmq driver has a direct dependency on eventlet, tests are
skipped under py3.

Co-Authored-By: Kapil Thangavelu <kapil.thangavelu@canonical.com>
Co-Authored-By: Edward Hope-Morley <edward.hope-morley@canonical.com>

Change-Id: I93b8b2e92d0f2a353d3357a5e61f6d472ec84944
Partial-bug: #1302941
2014-12-09 15:02:31 +00:00
Mehdi Abaakouk 15aa5cbda8 The executor doesn't need to set the timeout
It's up to the driver to set a suitable timeout for polling the broker,
this one can be different that the one requested by the driver
caller as long as the caller timeout is respected.

This change also adds a new driver listener API, to be able
to stop it cleanly, specially in case of timeout=None.

Closes bug: #1400268
Closes bug: #1399257
Change-Id: I674c0def1efb420c293897d49683593a0b10e291
2014-12-08 12:59:33 +01:00
Mehdi Abaakouk 43a9dc1de5 qpid: honor iterconsume timeout
The qpid driver must always honor the timeout passed the iterconsume
method, this change fixes that.

Related bug: #1400268
Related bug: #1399257

Change-Id: I8f267fc8b5a7abc852f0caf84d1e7c2c342ba951
2014-12-08 12:08:12 +01:00
Mehdi Abaakouk 023b7f44e2 rabbit: more precise iterconsume timeout
The iterconsume always set the timeout of kombu to 1 second
even the requested timeout more precise or < 1 second.

This change fixes that.

Related bug: #1400268
Related bug: #1399257

Change-Id: I157dab80cdb4afcf9a5f26fa900f96f0696db502
2014-12-08 12:08:01 +01:00
Jenkins a7f3ecb360 Merge "Touch up grammar in warning messages" 2014-12-05 09:58:33 +00:00
Jenkins d2105f61f8 Merge "Add more TLS protocols to rabbit impl" 2014-12-04 22:18:09 +00:00
Jenkins 1834167b4f Merge "Warn user if needed when the process is forked" 2014-12-04 21:47:03 +00:00
Doug Hellmann 66db2b310d Touch up grammar in warning messages
Change-Id: I556e6e646417be3732a7c31987ec964f9808f079
2014-12-04 16:04:37 -05:00
Jenkins bf9fd263f9 Merge "Fix reconnect race condition with RabbitMQ cluster" 2014-12-04 14:52:28 +00:00
Mehdi Abaakouk 712f6e3c5e Reintroduces fake_rabbit config option
This change reintroduces the fake_rabbit only for backward compatibility,
but mark it as deprecated.

Now, to use the kombu in-memory driver (that is not thread safe) we must
set the transport_url config option to 'kombu+memory:////" or the
rpc_backend to kombu+memory.

Or we can use the fake driver of oslo.messaging by setting the
transport_url to 'fake:///' or the rpc_backend to 'fake'

This is effectively reverting commit bcb3b23b8f.

Closes-bug: #1399085

Change-Id: I7b6fb3811fc6f695f75ecd350e04e69afd26c428
2014-12-04 11:54:58 +00:00
Jenkins 3828357c71 Merge "Rabbit: Fixes debug message format" 2014-12-02 16:00:55 +00:00
Mehdi Abaakouk cb78f2e43d Rabbit: Fixes debug message format
Add the missing 's' for the message format and
just print the number queues instead of print a list
full of '<object object at 0xXXXXXXXXXX>'.

Change-Id: Idaab4057bc6a41523a1944ae0c8e15e5e885b390
2014-12-02 15:32:34 +01:00
Mehdi Abaakouk 2dd7de989f Rabbit: iterconsume must honor timeout
The iterconsume method of the rabbit driver must
honor timeout parameter when reconnection to the broker
occurs.

Change-Id: I666d818449750c6bae9dde02f519842687a8e4fa
2014-12-02 14:42:04 +01:00
Mehdi Abaakouk bcb3b23b8f Don't use oslo.cfg to set kombu in-memory driver
This removes a TODO and also fixes a issue due to the
global state of oslo.config.cfg.CONF.

Closes bug: #1397339

Change-Id: Ib366f35678f899fda93821e6f07897baf8f631b4
2014-12-02 14:42:04 +01:00
Jenkins 42a2df15dd Merge "Have the timeout decrement inside the wait() method" 2014-12-02 11:10:50 +00:00
Mehdi Abaakouk f3370da11a Don't share connection pool between driver object
Each driver instance must use it's own connection pool.

This removes the last global state of qpid and rabbitmq driver
Make the relation between classes more simple.

The previous behavior was not very safe, as explained in the bug report.

And also, this is a first step to replace this custom connection pool
handling by the kombu one.

Closes bug: #1397925
Partial bug: #1397339

Change-Id: Iecd2b39c76417d9ac081d46810f72eb6e38edfda
2014-12-02 08:26:17 +01:00
Jenkins f2234d291f Merge "Show what the threshold is being increased to" 2014-12-02 07:05:38 +00:00
Joshua Harlow c7d99bf28f Show what the threshold is being increased to
Change-Id: I2182ca88126d636970885fe4403779a879f32aa7
2014-12-01 21:14:23 -08:00
Jenkins bd56131b58 Merge "Remove the use of PROTOCOL_SSLv3" 2014-11-28 08:09:13 +00:00
Joshua Harlow f1c7e78a56 Have the timeout decrement inside the wait() method
Currently it appears the timeout provided to the wait()
method will be reused X number of times (where X may be
indeterminate depending on reconnections, loss of messages
and so-on) so instead of reusing it which can potentially
result in a infinite number of calls have a new object be
used that will cause the subsequent timeouts used elsewhere
in the wait function to actually decay correctly.

Closes-bug: #1379394

Change-Id: I12c4ea1eef6b857d12246db0483adaf7c87e740c
2014-11-27 11:22:20 -08:00
Mehdi Abaakouk eb21f6b263 Warn user if needed when the process is forked
This change warns the library consumer when the process if forked and
we can't be sure that the library work as expected.

This also add some documentation about forking oslo.messaging Transport
object.

Change-Id: I2938421775aa72866adac198d70214856d45e165
Related-bug: #1330199
2014-11-27 16:29:20 +01:00
Jens Rosenboom 7ad0d7eaf9 Fix reconnect race condition with RabbitMQ cluster
Retry Queue creation to workaround race condition
that may happen when both the client and broker race over
exchange creation and deletion respectively which happen only
when the Queue/Exchange were created with auto-delete flag.

Queues/Exchange declared with auto-delete instruct the Broker to
delete the Queue when the last Consumer disconnect from it, and
the Exchange when the last Queue is deleted from this Exchange.

Now in a RabbitMQ cluster setup, if the cluster node that we are
connected to go down, 2 things will happen:

 1. From RabbitMQ side, the Queues w/ auto-delete will be deleted
    from the other cluster nodes and then the Exchanges that the
    Queues are bind to if they were also created w/ auto-delete.
 2. From client side, client will reconnect to another cluster
    node and call queue.declare() which  create Exchanges then
    Queues then Binding in that order.

Now in a happy path the queues/exchanges will be deleted from the
broker before client start re-creating them again, but it also
possible that the client first start by creating queues/exchange
as part of the queue.declare() call, which are no-op operations
b/c they alreay existed, but before it could bind Queue to
Exchange, RabbitMQ nodes just received the 'signal' that the
queue doesn't have any consumer so it should be delete, and the
same with exchanges, which will lead to binding fail with
NotFound error.

Illustration of the time line from Client and RabbitMQ cluster
respectively when the race condition happen:

       e-declare(E)      q-declare(Q)       q-bind(Q, E)
     -----+------------------+----------------+----------->
                               e-delete(E)
     ------------------------------+---------------------->

Change-Id: Ideb73af6f246a8282780cdb204d675d5d4555bf0
Closes-Bug: #1318721
2014-11-27 13:49:52 +01:00
Jenkins 6e4ccfa60c Merge "Create a new connection when a process fork has been detected" 2014-11-27 12:34:36 +00:00
Jenkins 0b839c57b7 Merge "Notification listener pools" 2014-11-27 11:56:42 +00:00
Kenneth Giusti 80e62aed7d Create a new connection when a process fork has been detected
This patch attempts to deal with applications that have forked the
process after connecting to the broker.  First, the creation of the
connection is delayed until the application attempts to perform a
messaging operation.  Second, each time the application performs a
messaging operation the current process id is checked against the id
of the process that created the connection.  If the process ids do not
match, the application has called os.fork().  The new child process
discards the existing connection and creates a new one.

Change-Id: I5455cb0f8d380d6b65f1268b34a91355cbb14aa2
Closes-Bug: #1392868
2014-11-26 21:24:54 -05:00
Brant Knudson 1624793088 Add more TLS protocols to rabbit impl
Python 2.7.9 added PROTOCOL_TLSv1_1 and PROTOCOL_TLSv1_2, so these
are added to the allowed kombu_ssl_version values.

See https://docs.python.org/2/library/ssl.html#ssl.PROTOCOL_TLSv1_1

Change-Id: I1dd590d916ab524284a941db91b9cb81fd4639bb
2014-11-25 10:59:05 -06:00
Thomas Goirand 42f55a1dda Remove the use of PROTOCOL_SSLv3
The PROTOCOL_SSLv3 should not be used, as it can be exploited with
a protocol downgrade attack. Also, its support has been removed in
Debian, so it simply doesn't work at all now in Sid.

This patch removes PROTOCOL_SSLv3 from one of the possible protocols
used by oslo.messaging.

Closes-Bug: #1395095
Change-Id: I2c1977c3bfc1923bcb03744e909f2e70c7fdb14c
2014-11-25 10:47:16 -06:00
Jenkins e67e9a6aa9 Merge "rabbit: uses kombu instead of builtin stuffs" 2014-11-20 23:19:33 +00:00
Jenkins bc237be16e Merge "Remove unuseful param of the ConnectionContext" 2014-11-20 11:53:10 +00:00
Jenkins 1a53d3ed30 Merge "Enable user authentication in the AMQP 1.0 driver" 2014-11-19 16:34:00 +00:00
Mehdi Abaakouk 973301aa70 rabbit: uses kombu instead of builtin stuffs
The rabbit driver have a custom code to reconnect to a broker,
to change the broker in HA configuration, to retry a to send
a message, to handle the interval between reconnection.

But all of that exists in kombu, so just use it.

Using the kombu Connection object with the url make also the rabbit
driver more generic.

Futher patches can rename rabbit* oslo.config options to a more generic
name and add a new driver entry_point 'kombu' to allow to use this driver
with any borker supported by kombu.

Change-Id: Id6b89d5448126ca652b46fe6ce5a9b3ed5839795
2014-11-19 17:21:02 +01:00
Elena Ezhova 0d49793e34 Create ZeroMQ Context per socket
ZeroMQ Context is a singleton and thus is created only once. This leads
to problems when there is more than one process working with it.
For example, while Neutron server starts, it firstly loads core
plugin and service plugins, which start message handling server,
and only then forks to create api-workers. As a result, all child
processes get the same copy of the context.

Creating new Context for each socket will prevent such situations
from happening and will guarantee that each process works with its
own Context.

Change-Id: I56912e39b119c20f6f23311fc2c7c4b9e9e480d0
Closes-Bug: #1364814
2014-11-18 18:18:07 +03:00
Mehdi Abaakouk 7306680bfa Remove unuseful param of the ConnectionContext
The ConnectionContext depends on the ConnectionPool.
A ConnectionPool already known the connection classes,
the configuration object and the url needed to create a new connection.

But we pass again thoses informations when we create a ConnectionContext.
This is unuseful, we can reuse thoses in the connection pool even we
want a not pooled connection.

This change removes the unuseful ConnectionContext parameters,
this also ensures that connection created with or without the pool
are created in the same ways and only at one place (the create method
of the connection pool).

Change-Id: I4bd43d202fa2774ad5dcb0f8dd05e58ba60c6009
2014-11-18 13:58:33 +01:00