Commit Graph

159 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov 115cfb5b7c Fix typo in quorum-related variables for RabbitMQ
In [1] there was a typo made in variable names. To prevent even futher
awkwardness regarding variable naming, we fix typo and publish a
release note for ones that already using variables in their deployments.

[1] https://review.opendev.org/c/openstack/oslo.messaging/+/831058

Change-Id: Icc438397c11521f3e5e9721f85aba9095e0831c2
2023-02-14 15:20:00 +00:00
Zuul 4186386748 Merge "Add quorum queue control configurations" 2022-06-13 17:14:16 +00:00
Zuul 64888bd05a Merge "Add a new option to enforce the OpenSSL FIPS mode" 2022-04-26 14:15:36 +00:00
hamza alqtaishat 8932ad237b Add quorum queue control configurations
the quorum queue type add features that did not exist before or not
handled in rabbitmq the following link shows some of them
https://blog.rabbitmq.com/posts/2020/04/rabbitmq-gets-an-ha-upgrade/

the options below control the quorum queue and ensure the stability of
the quorum system
x-max-in-memory-length
x-max-in-memory-bytes
x-delivery-limit

which control the memory usage and handle message poisoning

Closes-Bug: #1962348
Change-Id: I570227d6102681f4f9d8813ed0d7693a1160c21d
2022-04-06 19:46:40 +00:00
Hervé Beraud 7e8acbf870 Adding support for rabbitmq quorum queues
https://www.rabbitmq.com/quorum-queues.html

The quorum queue is a modern queue type for RabbitMQ implementing a
durable, replicated FIFO queue based on the Raft consensus algorithm. It
is available as of RabbitMQ 3.8.0.

the quorum queues can not be set by policy so this should be done when
declaring the queue.

To declare a quorum queue set the x-queue-type queue argument to quorum
(the default is classic). This argument must be provided by a client at
queue declaration time; it cannot be set or changed using a policy. This
is because policy definition or applicable policy can be changed
dynamically but queue type cannot. It must be specified at the time of
declaration.

its good for the oslo messaging to add support for that type of queue
that have multiple advantaged over mirroring.

If quorum queues are sets mirrored queues will be ignored.

Closes-Bug: #1942933
Change-Id: Id573e04c287e034e50626daf6e18a34735d45251
2022-02-05 07:12:49 +00:00
Hervé Beraud 384738a92d Add a new option to enforce the OpenSSL FIPS mode
This option ``ssl_enforce_fips_mode`` allow us to enforce the FIPS mode
if supported by the version of python in use.

https://en.wikipedia.org/wiki/Federal_Information_Processing_Standards

Change-Id: I50c7de71bfd38137eb83d23e910298946507ce9f
2021-11-08 15:05:30 +01:00
Hervé Beraud 4937949dff Correctly handle missing RabbitMQ queues
Currently, setting the '[oslo_messaging] direct_mandatory_flag' config
option to 'True' (the default) will result in a 'MessageUndeliverable'
exception being raised when sending a reply if a RabbitMQ queue is
missing [1]. It was the responsibility of the application to handle
this exception, however, many applications are not doing so. This has
resulted in a number of bug reports.

Start handling this error condition, using a retry loop to attempt to
resend the message and work around any temporary glitches. Since
attempting to send a reply will will no longer raise an exception,
there is little benefit in retaining the '[oslo_messaging]
direct_mandatory_flag' config option: users setting this to False will
simply not benefit from the retry logic and improved logging added
here. This option is already deprecated though and will be fully
removed in a future release.

[1] https://www.rabbitmq.com/channels.html

Change-Id: Id5cddbefbe24ef100f1cc522f44430df77d217cb
Closes-Bug: #1905965
2021-02-04 09:47:08 +00:00
Hervé Beraud 54f8724be2 Fix doc title rendering
Change-Id: I9532f28a84b206c6268ead94c574aacb6216df6a
2020-12-04 11:16:41 +01:00
Hervé Beraud a997f09e5b Adding pre-commit
Introduced changes:
- pre-commit config and rules
- Add pre-commit to pep8 gate, Flake8 is covered in the pre-commit hooks.
- Applying fixes for pre-commit compliance in all code.

Also commit hash will be used instead of version tags in pre-commit to
prevend arbitrary code from running in developer's machines.

pre-commit will be used to:
- trailing whitespace;
- Replaces or checks mixed line ending (mixed-line-ending);
- Forbid files which have a UTF-8 byte-order marker (check-byte-order-marker);
- Checks that non-binary executables have a proper
  shebang (check-executables-have-shebangs);
- Check for files that contain merge conflict strings (check-merge-conflict);
- Check for debugger imports and py37+ breakpoint()
  calls in python source (debug-statements);
- Attempts to load all yaml files to verify syntax (check-yaml);
- Run flake8 checks (flake8) (local)

For further details about tests please refer to:
https://github.com/pre-commit/pre-commit-hooks

Change-Id: Ibd0c3d64fdc5c293d9d676d33eab828d9fde971f
Co-authored-by: Moisés Guimarães de Medeiros <moguimar@redhat.com>
2020-09-22 12:35:37 +02:00
melissaml c2074e4760 Fix pygments style
New theme of docs respects pygments_style.

more info: http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014971.html

Change-Id: I287fd718f9283e5d7d6951b9e55d5a6b48e07e3c
2020-06-04 15:27:58 +08:00
Andreas Jaeger 9cc3f52ac3 Switch to newer openstackdocstheme and reno versions
Switch to openstackdocstheme 2.2.0 and reno 3.1.0 versions. Using
these versions will allow especially:
* Linking from HTML to PDF document
* Allow parallel building of documents
* Fix some rendering

Update Sphinx version as well.

Remove docs requirements from lower-constraints, they are not needed
during install or test but only for docs building.

openstackdocstheme renames some variables, so follow the renames
before the next release removes them. A couple of variables are also
not needed anymore, remove them.

Depends-On: https://review.opendev.org/728938
Change-Id: I70c7edf8b95cde890e6263195be1de6bb826e700
2020-05-18 20:55:51 +02:00
Hervé Beraud 7f4dafc327 Add release notes links to doc index
Change-Id: I93ae9330509b89629d8c360437065267db637f49
2020-04-16 15:30:22 +02:00
Hervé Beraud 32a1b6e948 [ussuri][goal] Drop python 2.7 support and testing
OpenStack is dropping the py2.7 support in Ussuri cycle.

Complete discussion & schedule can be found in
-
http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.html
- https://etherpad.openstack.org/p/drop-python2-support

Ussuri Communtiy-wide goal:
https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html

Change-Id: I15cc79159b603e232e442e202c78d6c56dc73252
2020-02-03 18:35:08 +01:00
caoyuan 1e7a0c6fbb tox: Keeping going with docs
Sphinx 1.8 introduced [1] the '--keep-going' argument which, as its name
suggests, keeps the build running when it encounters non-fatal errors.
This is exceptionally useful in avoiding a continuous edit-build loop
when undertaking large doc reworks where multiple errors may be
introduced.

[1] https://github.com/sphinx-doc/sphinx/commit/e3483e9b045

Change-Id: If9885a1f064226909181d8b69241eb814deb2105
2019-10-24 14:21:22 +08:00
Hervé Beraud b56380654a Introduce RabbitMQ driver documentation
Introduce a RabbitMQ driver documentation for admin.

Describing:
- some RabbitMQ and AMQP specifications (exchanges, queues, routing-key)
- the heartbeat specification and the type of used threads
- the driver options

Change-Id: I8fd1624834510f8dee81ab9342c708d726b8f827
2019-08-08 17:59:56 +02:00
Zuul dfc8fe9a8f Merge "Support kafka message compression" 2019-08-05 15:53:35 +00:00
Zuul a5276be3dc Merge "Bump the openstackdocstheme extension to 1.20" 2019-07-22 15:12:22 +00:00
Zuul 46cdd88358 Merge "doc: Cleanup admin docs" 2019-07-19 08:26:04 +00:00
pengyuesheng 33dc529c37 Bump the openstackdocstheme extension to 1.20
Some options are now automatically configured by the version 1.20:
- project
- html_last_updated_fmt
- latex_engine
- latex_elements
- version
- release.

Change-Id: Ib5e22f6a5374f05e576bbc00a209209fdb09acad
2019-07-19 11:24:08 +08:00
pengyuesheng 515a2d79ef Blacklist sphinx 2.1.0 (autodoc bug)
See https://github.com/sphinx-doc/sphinx/issues/6440 for upstream details
Depend-On: https://review.opendev.org/#/c/663060/

Change-Id: I648dd4d4a98a3d423c98d28c193737cffe1a2c49
2019-07-05 10:16:48 +08:00
Stephen Finucane f0eee1e16b doc: Cleanup admin docs
Use a sensible header style, fix some syntax highlighting, and generally
tidy things up.

Change-Id: I0b141b968ed8db10ff41a626569dd185edbdc641
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2019-07-04 14:50:57 +01:00
zhang-shaoman 9a752862e2 Support kafka message compression
When the message is large, in order to improve the efficiency of
kafka, we need to compress the message before send it, so we need to
support kafka message compression.

Change-Id: I9e86d43ad934c1f82dc3dcf93d317538f9d2568e
Implements: blueprint support-kafka-compression
2019-06-20 11:18:35 +08:00
Ben Nemec 70b03bcf83 Download kafka from archive.apache.org
It seems that versions are deleted from www.apache.org pretty quickly.
They stick around longer on archive.apache.org so we won't have to
be constantly chasing the latest version in our functional tests.

Change-Id: I047edac67699dd598f8dfd0f859b3772f6068bd3
2019-06-10 15:35:43 +00:00
Kenneth Giusti 4f467fbd73 Cap Bandit below 1.6.0 and update Sphinx requirement
Bandit 1.6.0 accidentally changed how the exclusion list option is
handled and breaks our use of it. Cap to the previous version until
Bandit has fixed the problem.

Sphinx 2.0 no longer works on python 2.7, so we need to start
capping it there as well.

Change-Id: Ie6b379f2c99862c37891ac03c52464e07bc2b2cc
2019-05-14 10:46:32 +02:00
jacky06 9b7bcbda2f Replace git.openstack.org URLs with opendev.org URLs
Change-Id: Ib4a523adf3275a8727d5a4f2a149e163a81be3cc
2019-04-26 10:31:26 +08:00
Andy Smith 0953fa1759 Kafka driver deployment guide
Depends-On: Idfb9fe3700d882c8285c6dc56b0620951178eba2
Change-Id: If8370c0c83312d675bde837f768ae40ec3603972
2019-01-23 08:27:43 -05:00
Andy Smith 5a842ae155 Switch driver to confluent-kafka client library
This patch switches the kafka python client from kafka-python to
confluent-kafka due to documented threading issues with the
kafka-python consumer and the recommendation to use multiplrocessing.
The confluent-kafka client leverages the high performance librdkafka
C client and is safe for multiple thread use.

This patch:
* switches to confluent-kafka library
* revises consumer and producer message operations
* utilizes event.tpool method for confluent-kafka blocking calls
* updates unit tests
* adds kafka specific timeouts for functional tests
* adds release note

Depends-On: Ice374dca539b8ed1b1965b75379bad5140121483
Change-Id: Idfb9fe3700d882c8285c6dc56b0620951178eba2
2018-12-04 11:25:07 -05:00
Zuul 7b6e2e49e3 Merge "doc: Remove crud from conf.py file" 2018-10-23 10:23:04 +00:00
Hervé Beraud 0410bcbf54 Using pip as a python module
Since python 2.7.9 and python 3.4
pip is a stdin module

Change-Id: I17c8c93337fa4f0a33f9131d4117739179133ea9
2018-10-15 10:56:55 +02:00
Hervé Beraud ad0d99c812 doc: Remove crud from conf.py file
Change-Id: I35c6221f78cac9fe4ab3e194e8a94ca4f0b24ca2
2018-10-15 09:42:18 +02:00
Andy Smith f2fd8d4375 Remove the deprecated ZeroMQ driver
Users of the oslo.messaging RPC communications service must use
the rabbit ("rabbit://...") or AMQP 1.0 ("amqp://...") drivers.

Change-Id: If3474142f1fe99d41d7b4466061ed0e23ca38549
Closes-Bug: 1789259
2018-08-28 16:00:50 -04:00
Kenneth Giusti 42f0358d5d Mark the ZeroMQ driver deprecated
As per the Dublin 2018 PTG decision:

http://lists.openstack.org/pipermail/openstack-dev/2018-March/128055.html

Change-Id: I1cda5dffbc29aad5fea001a79562db7c144a339b
2018-06-11 15:11:31 -04:00
Tovin Seven d1f241a410 Trivial: Update pypi url to new url
Pypi url changed from [1] to [2]

[1] https://pypi.python.org/pypi/<package>
[2] https://pypi.org/project/<package>

Change-Id: I1cee400bc9dce4f02235e519833b05bb0f4e500d
2018-04-20 17:12:43 +07:00
Kenneth Giusti 1231c4cbf2 Move requirements for the optional drivers (amqp1, kafka)
Move the requirements for the optional drivers to
test-requirements.txt and setup.cfg.  The default rabbitmq driver's
dependencies should be the only hard requirements for the base
package.

Leaving ZeroMQ deps unchanged for now as it will be removed in Stein

Change-Id: I19dd699ccf87e43202ccefb99258fbaa9ea17b7e
2018-04-16 10:38:07 -04:00
Kenneth Giusti 222a939361 Remove the deprecated Pika driver
It is recommended that all users of the Pika driver transition to
using the Rabbit driver instead.  Typically this is done by changing
the prefix of the transport_url configuration option from "pika://..."
to "rabbit://...".  There are no changes required to the RabbitMQ
server configuration.

Change-Id: I52ea5ccb7e7c247abd95e2d8d50dac4c4ad11246
Closes-Bug: #1744741
2018-03-21 10:58:23 -04:00
OpenStack Proposal Bot 008f58335a Updated from global requirements
Change-Id: I9500a0d27953881cfad24d5542356c539353d6a7
2018-03-15 07:49:08 +00:00
OpenStack Proposal Bot 579d5fe34d Updated from global requirements
Change-Id: I289f97bc931e865feea2beb7b78456f8020da8c6
2018-01-27 18:24:28 +00:00
OpenStack Proposal Bot 95e98a98f1 Updated from global requirements
Change-Id: Ieee5fa811e69e7ae4f597ce361319758d8030bde
2018-01-24 02:16:24 +00:00
Kenneth Giusti c7cc6d0742 Create doc/requirements.txt
For compliance with the Project Testing Interface as described in:

https://governance.openstack.org/tc/reference/project-testing-interface.html

Refer to:

http://lists.openstack.org/pipermail/openstack-dev/2017-November/124815.html

Change-Id: I3d8c70f66931c15247b530ff7e1c89bfc753446f
2017-12-18 15:09:20 -05:00
Ben Nemec 47d66d2b46 Fix typo in contributor docs title
"Contributing oslo.messaging" is not grammatically correct.  It
should be "Contributing to oslo.messaging".

Change-Id: I71737086745beb6d8d76669ab04b7580f1b7b11a
2017-10-02 23:52:24 +00:00
Jenkins dc7bd0591b Merge "Add licenses and remove unused import in doc/source/conf.py" 2017-09-05 15:32:27 +00:00
Andrew Smith 8203b9ed73 Update amqp 1.0 driver deployment guide
This patch:
* Adds new options recently added to the driver
* Updates the devstack support information

Change-Id: I772e01ebb467a64c27b90e6105adf294bcb22654
2017-08-23 14:21:33 -04:00
liuyamin 59c0c8baa5 Add licenses and remove unused import in doc/source/conf.py
Change-Id: I1bded5759442c3d6dc59729f79f8166b999c9c60
2017-08-18 01:15:07 +00:00
Adam Spiers 1038a1c038 fix 'configration' typo
Fix typos in the documentation and corresponding directory structure.

Change-Id: I90c40139f6c6af3d326f6f9251e6821d5c4b1a82
2017-07-31 11:28:28 +01:00
Dong Ma bff426f0eb switch from oslosphinx to openstackdocstheme
Change-Id: Ic3d9b9cbef0040512e37c8e99df7bb028e344bc9
2017-07-06 15:50:11 +00:00
Dong Ma 280cb368f7 rearrange content to fit the new standard layout
Change-Id: I4dd1c9c289d31077b71f3d96877b776d9d5e16f7
Depends-On: Ia750cb049c0f53a234ea70ce1f2bbbb7a2aa9454
2017-07-06 03:19:38 +08:00
ZhangHongtao 53b699b90a Fix html_last_updated_fmt for Python3.
html_last_updated_fmt option is interpreted as a
byte string in python3, causing Sphinx build to break.
This patch makes it utf-8 string.

Change-Id: Ifde17e94d0345a289eea29ba3d664d31f1eb51d9
Closes-Bug:#1693670
2017-06-09 17:38:47 +08:00
Kenneth Giusti c35fb276ce Mark the Pika driver as deprecated
The Pika driver was intended to be a more stable and performant
replacement for the default rabbit driver.  However due to lack of
both maintainers and compelling evidence that pika is superior to the
existing rabbit driver in either performance or stability it has been
deprecated for removal.

See:
http://lists.openstack.org/pipermail/openstack-dev/2017-May/116679.html

Change-Id: I98e0123edd3248be665325833283689fc3a897f7
2017-05-30 13:21:23 -04:00
Andrew Smith ec4d6639bc Add get_rpc_transport call
The get_rpc_transport wraps get_transport to unify the API in
anticipation of comprehensive separation of RPC and Notification
messaging backends.

Related-Bug: 1680192
Change-Id: Ic6af07b98ff43806c2af38a3ba129991f1e0ec86
2017-05-12 12:30:41 -04:00
Stephen Finucane 1251c0fef1 Use Sphinx 1.5 warning-is-error
With pbr 2.0 and Sphinx 1.5, the setting for treat sphinx warnings as
errors is setting warning-is-error in build_sphinx section. Migrate
the setting from the old warnerrors one.

The history document is removed because some of the commit messages used
in ChangeLog were being identified as invalid markup and there doesn't
appear to be anyway to edit there retroactively nor disable warnings on
a specific file.

Change-Id: I79e7ac56d5af1151865686761f3d40a11efbf472
2017-03-16 15:22:18 +00:00