This function is no longer used since we removed ampq1 tests, when we
deprecated the amqp1 driver[1]
[1] 0f63c227f5
Change-Id: I47fe04d6a39ed2b5f33b02fa6736d588d0383f5a
The latest release of qdrouterd on focal has changed where the
internal python modules are installed. This patch updates the tox
tests python path configuration.
Change-Id: Icb53ee17af01580d899f388f69be9560e23675e0
'skip_basepython_conflicts' has been the cause of a couple of bugs in
tox 4 and there is talk of it going away. Remove it and fix up a few
other issues in the tox.ini file.
Change-Id: Ic19c896af2ab0cf3570c43e8ceb8cba64fb45cdd
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We already expose functions to handle the instantiation
of classes such as RPCServer and RPCTransport but the
same was never done for RPCClient so the API is
inconsistent in its enforcement.
This adds a get_rpc_client function that should be used
instead of instatiating the RPCClient class directly to
be more consistent.
This also allows to handle more logic inside the function
in the future such as if implementations for an async client
is implemented, as investigation in [1] has shown.
[1] https://review.opendev.org/c/openstack/oslo.messaging/+/858936
Change-Id: Ia4d1f0497b9e2728bde02f4ff05fdc175ddffe66
A recent oslo.messaging patch [1], not yet merged, who aim to update the
test runtime for antelope lead us to the following error:
```
qdrouterd: Python: ModuleNotFoundError: No module named 'qpid_dispatch'
```
Neither debian nor ubuntu in the latest releases have any binary
built for the qpid backend, not even 3rd party. Only qpid proton,
the client lib, is available.
To solve this issue, these changes propose to deprecate the AMQP1 driver
who is the one based on qpid and proton, and propose to remove the
related functional tests.
The AMQP1 driver doesn't seems to be widely used.
[1] https://review.opendev.org/c/openstack/oslo.messaging/+/856643
Closes-Bug: 1992587
Change-Id: Id2ca9cd9ee8b8dbdd14dcd00ebd8188d20ea18dc
Introduced changes:
- pre-commit config and rules
- Add pre-commit to pep8 gate, Flake8 is covered in the pre-commit hooks.
- Applying fixes for pre-commit compliance in all code.
Also commit hash will be used instead of version tags in pre-commit to
prevend arbitrary code from running in developer's machines.
pre-commit will be used to:
- trailing whitespace;
- Replaces or checks mixed line ending (mixed-line-ending);
- Forbid files which have a UTF-8 byte-order marker (check-byte-order-marker);
- Checks that non-binary executables have a proper
shebang (check-executables-have-shebangs);
- Check for files that contain merge conflict strings (check-merge-conflict);
- Check for debugger imports and py37+ breakpoint()
calls in python source (debug-statements);
- Attempts to load all yaml files to verify syntax (check-yaml);
- Run flake8 checks (flake8) (local)
For further details about tests please refer to:
https://github.com/pre-commit/pre-commit-hooks
Change-Id: Ibd0c3d64fdc5c293d9d676d33eab828d9fde971f
Co-authored-by: Moisés Guimarães de Medeiros <moguimar@redhat.com>
test-setup.sh is called after bindep is run in CI, and tox is not setup.
Simplify the script so that it only does what's needed - install
qdrouterd for Ubuntu from qpid/testing.
Install the other packages using bindep, add test to amqp1 profile.
Add amqp1 to bindep_profiles where needed for scenario tests, previously
test-setup.sh added it everywhere.
Some tests are skipped due to changes in recent versions of kombu. These
will be unskipped when we come up with a proper fix.
Change-Id: Ic3a9e2c873619670edfbf71022d593f3cb5f70f2
Related-Bug: #1885923
Remove six, the python 2/3 compatibility library. It's not needed
anymore since the repo is python3 only.
Remove a now unneeded hacking test.
Change-Id: I40522c4accb4aaf8115d11fee8b081e2d991cb4d
This patchset introduces scenarios for the functional tests to unify the
setup and configuration of alternate messaging backends for RPC and
Notifications. The scenarios are defined to reduce redundancy in
the testing of backends and to demonstrate functional correctness
across driver combinations.
Current driver support:
rabbit - RPC, Notify
amqp - RPC
kafka - Notify
RPC Notify
--------- ----------
scenario01 rabbit rabbit
scenario02 rabbit kafka
scenario03 amqp rabbit
scenario04 amqp kafka
It is anticipated that additional scenarios will be defined as new
drivers are introduced and/or new messaging intermediarites are supported.
Note: The removal of python2 jobs are also included patch
Change-Id: I0f86416623a0b718516147f0660b4df2b74cf867
Without this fix, the simulator splits the "topic" value in different
strings, in notify-client case
Instead of having one queue there are multi-queues.
See [1] for more details
Closes-Bug: #1824339
1- https://bugs.launchpad.net/oslo.messaging/+bug/1824339
Change-Id: I438c7279d584bdb00da18477a83d5e2a53c588c4
This patch messaging intermediaries used for the amqp1 driver
test. The combination of the apache qpid-dispatch-router (qdrouterd)
with an attached apache artemis broker is used in place of the qpidd
broker. The qdrouterd will directly message oslo.messaging rpc
communications and will link route oslo.messaging notify
communications to the artemis broker (e.g. for persistence).
This patch:
* updates the dependencies
* updates the test environment setup
* uses the pifpaf drivers to create servers
Change-Id: I9fb56450346fca84bc1573fa5e9b6fe81c14e33a
This patch addresses a number of issues that prevented the functional
tests from running. The functional tests now execute and can complete
succesfully. At times, the test will fail (noticiably in CI) indicating
an underlying issue with consumer interaction with the kafka server.
It would be beneficial to merge this patch as it provides repeatability
and visibility for driver-kafka server integration to facilitate
additional debugging and testing.
This patch:
* removes use of deprecated get_transport
* override consumer_group for each test
* changed to synchronous send
* update to kafka 1.0.0 server
Depends-On: Ib552152e841a9fc0bffdcb7c3f7bc75613d0ed62
Change-Id: I7009a3b96ee250c177c10f5121eb73d908747a52
We do not need tox_install.sh, pip can handle constraints itself
and install the project correctly. Thus update tox.ini and remove
the now obsolete tools/tox_install.sh file.
This follows https://review.openstack.org/#/c/508061 to remove
tools/tox_install.sh.
Change-Id: Ifaaf656effff20ef08214f111645a3b5fc8b4d28
Oslo.messaging uses a single bindep.txt to define common
and driver dependencies. The bindep_profile is used by the
bindep role to select the driver specific packages.
This patch:
* Add bindep_profile var to jobs
* Add kafka dependencies
* update test-setup for use by amqp1
Depends-On: I6beb7b07a7dbc345e63e758ebbc86400e0ee16d3
Change-Id: Ib552152e841a9fc0bffdcb7c3f7bc75613d0ed62
- kombu4 wraps recoverable errors as OperationalErrors rather than
raising amqp errors
- also, raise a recoverable error and redeclare if for some reason a
message is double acknowledged... previously, this was hidden.
- ensure socket is not none
- use connect method to ensure connection
Depends-On: I9f980b51901ac31599b9651633956ad2eea6a1ac
Change-Id: I73958c8057353a2eefe1baaa7a41148193d507f7
Currently Kafka driver for an oslo.messaging uses kafka-python==0.9.5
and mostly broken. This package version supports only low level Kafka
producer and consumer API which are marked as deprecated now [1]. Using
of these interfaces bring a big concern to the message processing,
because current KafkaConsumer has not any consuming coordination. This
fact causes a message duplication for the several consumers of one
topic. This behavior is specific to Ceilometer and services which read
and process notifications from other services.
New version of kafka-python allows to use async thread safe message
producers and coordinated consumers [1].
[1] http://kafka-python.readthedocs.io/en/master/changelog.html#feb-15-2016
The driver is currently experimental, python-kafka<1.0.0 API have major
issue described above that can't make the oslo.messaging driver works,
so we prefer having a working driver with a non-synced dependencies, that the
reverse.
Co-Authored-By: Mehdi Abaakouk <sileht@redhat.com>
Change-Id: I29862ed7bf56b9d8878fa8e9fb1cbd9d643908a4
Adding constraints support to libraries is slightly more complex than
services as the libraries themselves are listed in upper-constraints.txt
which leads to errors that you can't install a specific version and a
constrained version.
This change adds constraints support by also adding a helper script to
edit the constraints to remove oslo.messaging.
Change-Id: I8be883215f27abb58d15b85e8542cbdf32000bac
1) Remove hardcoding 'redis' matchmaker from any zmq:// transport because it's
fails in cases zmq+etcd:// or zmq+sentinel:// urls.
2) Allow to use transport_url from config file by removing default value for
args.url option. In this case by default args.url is None and
transport_url will be used from config file.
Change-Id: I1edcabe46b1e92dd4e6eccc9151b50d03f394186
Do initialization of the generator only for rpc/notify clients.
No needs to load messages_length.yaml and calculate distribution for servers.
Change-Id: Ib1ac30181bec20eacf6d29ccd5f76c0b6b1e96f8
This patch provide the sync flag:
simulator.py rpc-client --sync call
simulator.py rpc-client --sync fanout
The --sync values means next:
- call: to send sync msg via rpc.call
- fanout: to broadcast sync msg via rpc.fanout to all servers on topic
When clients has sent all messages, the rpc-server will be stopped
automatically and dump statistics to a file (if --json-file is used).
This is much usefull than rough killing the process of the server in
benchmark frameworks.
Change-Id: I06fd8dbbcdc8b2b9f13029029f730b417ff128ce
--debug, --is-cast and --is-fanout are defined as args of type=bool.
This means that, for example, if we want to enable debug logging
level, we have to type '--debug True'. But we can also use
'--debug False' in order to do the same, which is very misleading
(in fact, any non-empty string will evaluate to True). This patch
tries to solve this problem by replacing type=bool and
default=False with action='store_true' for these args, so that
we will be able to enable them (they will remain False by default
as before) simply as '--debug' etc.
Change-Id: I8ee04c35427df446966161491da8d264b44975bf
When we calculate metrics like msg/sec, latency, etc
we expect the start and end time as time of
the first and last processed message:
RPC Server life timeline:
[----0..5 sec----][---5..10 sec---][---10..15 sec--]
waiting clients 10 msg recved wait sigint
expected: duration 5 sec, 2 msg/sec
actual (incorrect): duration 15 sec, 0.6 msg/sec
no reason to set the boundaries if server was idle few seconds
before running of clients and after.
Change-Id: I33e0a605b54ea7b89977504892528c41c3b00a68
ZeroMQ driver options are current stored into the DEFAULT group.
This change makes the zmq configuration clearer by putting its
options into oslo_messaging_zmq group.
Change-Id: Ia00fda005b1664750d2646f8c82ebdf295b156fb
Closes-bug: #1417040
Co-Authored-By: Oleksii Zamiatin <ozamiatin@mirantis.com>
Now it is possible to stop simulator client and server
by sending SIGINT or SIGTERM signals. Note that both stop
gracefully and it takes some time to do this.
Change-Id: Ie3fd1ea8b146070d61a247fd8ccc124df8d34848
Right now simulator.py expects messages_length.yaml to be in the
current directory and as a result, it is impossible to launch
simulator while you are at the repository root.
The change makes simulator to search for the file in the same
directory as simulator itself is in.
Change-Id: Ic2ca21d87e2d888432dc6204c7b4b995a06aecbd
Aligning message sending and stats collection to whole seconds.
This allows to merge data from client and server - useful
for visualization of message flow.
Change-Id: I507c021d851254d4d84e8922de687931f8545864
This patch introduces new parameter --json <file name>.
When it is specified the simulator stores stats in JSON file.
Change-Id: I0b7a87ee614f44baa920569fe0c7046269c90925
Takashi Kajinami