Commit Graph

85 Commits

Author SHA1 Message Date
Martin Chacon Piza 873cbf82f6 Rename tenant with project_id
Since oslo.context=4.0.0 Feb 3, 2022  the deprecated argument
tenant was removed [1]

- Bump docutils in lower-constraints and test-requirements

[1] f4cbe4c296

Change-Id: I703d49f5e5870bf45030fc5502749e197aae3a3b
2022-04-06 11:04:08 +02:00
Ghanshyam Mann c491b34888 [goal] Deprecate the JSON formatted policy file
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: Ibb26169656785330e7d86a362fd110222fb63a07
2021-02-02 17:44:35 +00:00
wangzihao 6ffab589a4 Remove six
Remove six Replace the following items with Python 3 style code.
- six.PY3
- six.xrange
- six.string_types
- six.binary_type
- six.iteritems
- six.unichr
- six.wraps
- six.get_function_code
- six.PY2
- six.b
- six.moves.range
- reraise
- six.text_type

Story: 2008305
Task: 41301
Change-Id: Idf64154a013b53c7db771a25f1c63c1295e354e6
2020-11-23 15:20:59 +08:00
Martin Chacon Piza e8a953ba3a [goal] Migrate testing to ubuntu focal
As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).

-Bump the requirements and lower constraints for required deps
which added python3.8 support in their later version.

-It also fixes one unit tests to handle PEP479 being enabled for all code
in Python 3.8 [2].

Story: #2007865
Task: #40197

[1] https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.html
[2] https://stackoverflow.com/questions/51700960/runtimeerror-generator-raised-stopiteration-every-time-i-try-to-run-app

Change-Id: If67bbfe8a2db19fa8f3437f746ca7f4d04487447
2020-10-13 16:09:31 +02:00
Hervé Beraud a1593cd0ff Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: I691d3717750c7ba5084cac98fad5be22170732db
2020-06-02 20:25:01 +02:00
Andreas Jaeger 2ebc766534 Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found.

Remove hacking and friends from lower-constraints, they are not needed
for installation.

Change-Id: Ia16e8cbccf7b38be8909aa94e66c04b99f200b89
2020-04-24 12:41:05 +00:00
Sean McGinnis f38a954fd2
Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: Ie775b99cdb645f072c0a22fa1a1a47448970b103
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-04-18 11:54:48 -05:00
Martin Chacon Piza 0f4e2bc5df Replacing ujson with simplejson
- http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012285.html

Change-Id: Ic18f8cf4addfedd9055999ec685e81140ffda32d
2020-04-02 15:44:25 +02:00
liushuobj 0447757f4e fix a typo
Change-Id: I4e986887602ae17b1c90e49ada4cfe95f6cd15ae
2019-12-27 10:08:11 +08:00
Zuul 6ff39f3a4d Merge "Enable running unit tests in py37 environment" 2019-12-10 20:18:57 +00:00
Witek Bedyk 231f3b4760 Enable running unit tests in py37 environment
Until now no unit tests have been executed in py37 environment. This
change fixes tox configuration to enable running these tests.

It also fixes two unit tests to handle PEP479 being enabled for all code
in Python 3.7 [1, 2].

[1] https://docs.python.org/3/whatsnew/3.7.html#changes-in-python-behavior
[2] https://stackoverflow.com/questions/51700960/runtimeerror-generator-raised-stopiteration-every-time-i-try-to-run-app

Change-Id: I7fcf30d848efb8dc4cc6ca19056868aa6127cf60
2019-11-27 09:19:36 +01:00
Witek Bedyk eb8e6ff7e2 Allow passing config options to Kafka producer
The change allows passing supported options as dictionary to create a
Kafka producer object.

Change-Id: I7627a8caa2d6bb9c7789df143e72a4085060b164
Story: 2006059
Task: 37531
2019-11-18 11:06:12 +01:00
Witek Bedyk 66754b1a4a Avoid value decoding in Kafka delivery report
In Python 2 Kafka message value gets implicitly converted to unicode in
delivery_report function. In case the message value already contains
unicode characters the convertion fails with UnicodeDecodeError.

This case can be reproduced when trying to run:

  u'goose: {}'.format('gęś')

This commit fixes the debug message and avoids implicit type convertion.

Change-Id: Ia9463cd4e9f92cdf18b092abd2cf676aa4966679
Story: 2006503
Task: 36481
2019-09-12 14:08:12 +02:00
Zuul a6843fdd3b Merge "Confluent Kafka driver broker option bug" 2019-08-12 15:32:04 +00:00
Zuul a834fab114 Merge "Add LegacyKafkaMessage and Kafka client factory" 2019-08-12 14:15:17 +00:00
zhangjianweibj 2247194f08 Confluent Kafka driver broker option bug
In confluent Kafka driver, brokers option is string type not list type.
Comma separated list of host/port pairs. See
https://github.com/confluentinc/confluent-kafka-python/blob/master/README.md

Story: 2003705
Task: 36199

Change-Id: I5961ee32c91a9a946831a36dc88413e56b251e1e
2019-08-10 16:29:14 +02:00
Zuul 2a6cc6cc0a Merge "Support kafka connection timeout option" 2019-08-09 15:45:58 +00:00
Witek Bedyk bed8b9cd9f Add LegacyKafkaMessage and Kafka client factory
The change moves LegacyKafkaMessage object and client factory module
from the commponents commonly using it (persister, notification).

Story: 2003705
Task: 36094

Change-Id: I61ecf2cdf1dbe9a134df53c3a36614f84e8baa4e
2019-08-09 10:37:03 +02:00
Witek Bedyk 875c233fd7 Fix Confluent Kafka consumer topics subscription
The change fixes:

`TypeError: on_revoke expects a callable`

when `repartition_callback` is not set.

Change-Id: I488d6c99e2f2ad857d44b471c5b62e549062a46e
Story: 2003705
Task: 36093
2019-08-06 14:53:03 +02:00
zhangjianweibj 0b818510e3 Support kafka connection timeout option
Add timeout option for kafka consumer client.

Task: 36034
Story: 2006312
Change-Id: I197b81b2c5690b3e27b3fac8dd42c4cb2d9ba776
2019-07-31 08:34:37 +00:00
Witek Bedyk ce053d1094 Fix Confluent Kafka consumer loop
Even when there is no new message consumed the consumer loop should
proceed to execute the commit callback. Otherwise the batched messages
which which were consumed earlier do not get commited until new message
arrives. Although the case is nearly impossible to happen in real
environment it causes tempest tests to fail when only two messages are
created for testing.

Story: 2003705
Task: 35855

Change-Id: Iffee218b0a9cac572b6ea13d96da5b8f9e0d0b0d
2019-07-18 12:16:24 +02:00
Martin Chacon Piza 94bca491ed Removed unused code
The code of alarm_expr_parser was copied from monasca-api to monasca-common
in order to be used in monasca-ui, but after more than a year it isn't used
in monasca-ui.

Change-Id: I4b4dc0410715ab9e1f4c7fc786df10ec2f615b59
Story: 2001837
Task: 35800
2019-07-12 10:17:44 +02:00
Sumit Jamgade afc64e7de1 In Python 3.7 "async" and "await" will become reserved keywords
This patch changes the use of "async" with the PEP8 recommended
convention. As of Python 3.6, the following deprecation warning is
displayed:

See PEP 492 for further details.:
https://www.python.org/dev/peps/pep-0492/

Change-Id: I0e3fd5b70fb34ee3b4f1eb444c27279479b3bc60
Story: 2006176
Task: 35700
2019-07-08 14:23:33 +02:00
Witek Bedyk 5bdd98662b Catch the case when no new messages are available
Confluent Kafka client throws the KafkaError._PARTITION_EOF exception
when no new messages are available in a given partitiion. We should
catch this case and continue consuming after a short sleep.

Story: 2003705
Task: 30116
Change-Id: I44add24df764bbc4e718358a8af75903e035f3f0
2019-03-20 12:48:46 +01:00
Zuul 70d9aff4ca Merge "Add wrapper for Confluent Kafka client consumer" 2019-02-28 22:18:01 +00:00
Zuul dda9a87e50 Merge "Add the metric name to the Exception message when validation fails" 2019-02-28 16:30:20 +00:00
Witek Bedyk c542d908f1 Add wrapper for Confluent Kafka client consumer
This change provides a wrapper around a high-level Kafka consumer which
can be used in all Monasca components, in particular in
monasca-persister and monasca-notification.

Story: 2003705
Task: 29711
Change-Id: Iee317c3016f9868616022389209e36994cf185cf
2019-02-28 12:48:38 +01:00
Joseph Davis 5c6dddfc6e Add the metric name to the Exception message when validation fails
In the general validate_metric case, when possible add the metric
name to the exception message. This makes it much easier to
track down what metric failed.

Also, fixing code where an exception could be thrown but then
immediately ignored and another exception fo the same type
with a different message would be thrown.

Change-Id: I8b8f04bb98a65b6904894dc00af88ea367a350f8
Story: 2004185
Task: 27675
2019-02-28 10:37:10 +00:00
Martin Chacon Piza 0a1822a088 Improve coverage
Change-Id: I556791fc240e6f5fb8e0b5ddf9238294835a5015
Story: 2003881
Task: 29453
2019-02-18 09:41:59 +01:00
Witold Bedyk a8b68cbc19 Add wrapper for Confluent Kafka async Producer
Story: 2003705
Task: 27630
Change-Id: I5981e40ecf62cd123ff1969109a8b4b0b9ec204d
2018-11-28 09:56:34 +01:00
Witold Bedyk 1017229c39 Fix coverage job for Python 3
* add coverage check job to Zuul
* migrate to stestr as described in PTI [1]
* unmock OffsetOutOfRangeError to correctly recognize it as Exception
and not a mock. Otherwise TypeError exception is thrown [2].

[1] https://governance.openstack.org/tc/reference/pti/python.html#coverage-jobs
[2] http://logs.openstack.org/60/606660/2/check/openstack-tox-cover/b3c1492/testr_results.html.gz

Change-Id: Idcb8b11778ce02596c1c02417ee6c88a33c7da03
Story: 2003240
Task: 26900
2018-10-05 10:09:06 +02:00
melissaml 725d34c429 Update the outdated URL
Change-Id: I9be4cfb38f78e37cf3efadccd1e9eee2797a1bf5
2018-09-23 16:41:40 +08:00
Adrian Czarnecki 304772fa18 Ensures that messages pass to kafka client are bytes string
Story: 2003178
Task: 23324

Change-Id: I3ae6830fbbe80bf56726886d658c55dda965d717
2018-07-31 12:09:01 +02:00
Adrian Czarnecki 1cce6824ed Py3:Fix messages encoding for kafka producer
Story: 2000975
Task: 23056

Change-Id: Id0273c6c6f54342286d02e326a392e1479a15fe8
2018-07-23 13:12:17 +00:00
Adrian Czarnecki 38e3b64d57 Python3.5: Make KafkaProducer compatible with Py35
Change-Id: I9727a4aa4a2cabc1d55e6d242980152ef7d60011
2018-06-15 08:52:20 +02:00
Doug Hellmann 2a362bea97 ignore python2-specific code under python3 for pep8
The type 'long' no longer exists under python 3. The runtime execution
of the code using it in the metrics module already detects the python
version, but when the linter is run under python 3 it does not apply
the same check. Add a noqa pragma to tell the linter to ignore the
line where 'long' is referenced to avoid an error.

Change-Id: If806e6461358ea523708eceed6fdcc3dfb8d75a9
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-06-06 17:03:33 -04:00
zreigz 086b9efcaf Add alarm expression parser
Change-Id: I197d49a6b27d9f02b5e9b10359051052112c2a15
Story: 2001837
Task: 12601
2018-04-12 13:38:30 +02:00
Amir Mofakhar 806a418ede Updated pep8 checks
* set the maximum line length to 100
* removed unnecessary ignores
* updated codes for pep8
* added apache 2.0 license for missed files

Change-Id: If9398a8fdd094e50d6a07cb73f7389558dbe0a0d
Signed-off-by: Amir Mofakhar <amofakhar@op5.com>
2018-04-04 14:39:41 +00:00
Witold Bedyk 5f3bcbd50b Add messages for testing unicode
These messages can be used for testing unicode in monasca-log-api and
monasca-tempest-plugin.

Change-Id: I51843dcda5059599a87e3e136a75faa25da67660
Story: 2001400
Task:  6222
2018-01-29 15:30:53 +01:00
Amir Mofakhar 4250756a1f Implemented a policy enforcement engine
Under monasca_common/policy an enforcement engine is added for using
oslo.policy in the other monasca projects.
There are same methods as in enforcement engings currently used in
nova and keystone projects.
Also added unit tests under tests/policy for testing implemented
methods.

Task: 6105
Story: 2001233

Change-Id: Ic5402ba0986416c9386c1dc3fc1559f148ea9625
Signed-off-by: Amir Mofakhar <amofakhar@op5.com>
2017-12-19 15:16:09 +01:00
Craig Bryant 998f6afa53 Retry publish once on failures
On failure to publish, clear the topic metadata then retry, in
case the IP Addresses have changed.  This can occur when
Monasca is run in Kubernetes and the Kafka pod is restarted.
Restarting the Kafka pod can happen often enough the
API should be able handle it without losing a message.

Change-Id: If48971c40883b5be10adec947562cdda7e82d77c
Story: 2001386
Task: 5963
2017-12-12 12:47:27 +00:00
Ryan Brandt 7f91a53128 Allow () in dimensions
Change-Id: Ic61a325162873572cad387593c7107cba1a41e15
2017-11-30 15:24:46 -07:00
Habeeb Mohammed 94e0a059b5 Fixed inconfigurable kafka consumer offset location
PROBLEM: Consumer offset was resetting to the latest index rather than the earliest
SOLUTION: Modified consumer creation to include `auto_offset_reset="smallest"` which
	allows the offset to reset to the earliest known index.

NOTE: This does exactly what the whence parameter in SimpleConsumer.seek()
	is expected to do, however in order to achieve this functionality,
	the parameter `auto_offset_reset` MUST be set to either "largest"
	or "smallest".

Change-Id: I887892d80f2da9619c7f11737b3ab2e1d1dacf1e
2017-09-27 14:35:00 -06:00
Jenkins 9188cfcb34 Merge "Add a query language for group, inhibit, and silence rules" 2017-06-19 17:19:03 +00:00
Andrea Adams 41800dd195 Add a query language for group, inhibit, and silence rules
The new alarm rules will each have an expression in their
definition which will need to be parsed by both the Monasca-
API and the Monasca-Notification-Engine. Documentation for
this will be included in the API along with descriptions of the
new rules.

Story: 2000939
Task: 4692

Change-Id: I1a98fafae8dfdfa6fdb2eb66f4a4a4f40e518e46
2017-06-16 22:42:20 +00:00
Emma Foley a6648264cc [doc] Fix sublist formatting in monasca_common/kafka_lib/NOTES.md
Change-Id: I465e3fd86b7875e51f5d9b0a82e106ede6787390
Closes-Bug: #1696786
2017-06-08 15:25:50 +00:00
M V P Nitesh 03fd5ca884 Replace assertRaisesRegexp with assertRaisesRegex
This replaces the deprecated (in python 3.2) unittest.TestCase
method assertRaisesRegexp() with assertRaisesRegex()

Change-Id: I0bed1f2a0bb8ef57a48e3b778795e8ac75f3a2eb
2017-05-22 14:42:11 +05:30
Tomasz Trębski cabc2ddd5f Migrate to ostestr framework and PY3
Following commits does several things:

* migrates CI of monasca-common to ostestr
* enables PY35 compatybility

Also:

* marked one tests as excluded under PY35 because changing
it would require affecting embedded kafka library which
will be eventually removed in future

Change-Id: I432a466e2620bc8d305ef2630307b636461c8e81
2017-03-02 19:51:04 +01:00
Tomasz Trębski dc7c14d2d0 Removed unused logging module
monasca_common.logging seems to be unused
in other monasca projects therefore should
be removed.

Also removed one dependency that was used only
by that module

Change-Id: Ib875d9bae86c9b2b715edbe0226347b3fc9ec8ed
2017-02-17 05:50:51 +00:00
Tomasz Trębski 9cbcfed83f Use long as possible timestamp value
long, as type, is also possible to be a timestamp
format

Needed-By: I2f9d22a2c5e18826c8f9bb1e817ad963731b390f
Change-Id: I186abe4cdafd58d998f8aaf36d866795771a9e0a
2017-02-15 06:58:03 +01:00