Commit Graph

397 Commits

Author SHA1 Message Date
Hasan Acar a24fd834d0 fix: migrate CI to jammy
fix: change tox.ini

fix: change queries for list_dimension_names and list_dimension_values because of influxdb time filter problem

fix: remove build_sphinx group from setup.cfg

fix: handle hashlib security problem

Change-Id: I0d31a8db5ed71c70e7b878ce5e7940e041d0fa43

Change-Id: I6f7066da10e834550cbf0c053c7bf425ac0ead93

Change-Id: If9575aee73d600bbc84fcdf58deb1c57b508d9c2

Change-Id: If515eaeee7539da3ca49997e88785dc65572b334
2024-02-22 14:35:03 +00:00
Martin Chacon Piza 18ba158e5a Rename the deprecated argument tenant with project
Since oslo.context=4.0.0 Feb 3, 2022  the deprecated argument
tenant was removed [1]

- Remove temporarily monasca-tempest-log-python3-influxdb check
- Remove permanently monasca-tempest-python3-cassandra check

[1] f4cbe4c296

Change-Id: I38dc75df3e7a7575be37dbfa3714e694a19f35c4
2022-03-18 12:19:29 +01:00
Adrian Czarnecki 0b63819ff1 Fix problem with pyparsing.operatorPrecedence method
Pyparsing library was recently updated in golab requiremnts [1].
Since version 3.0.0 operatorPrecedence method was renamed to infixNotation [2].

[1]62f92c0187
[2]16b766b97c/CHANGES (L598)

Change-Id: I3bfefe5b9bc601f383e0b9d80046de387e420fd8
2021-12-15 15:28:26 -08:00
Martin Chacon Piza 73e942cc2e Fix create_notification unittest and bump lower-constraints
- The method create_notification returns an uuid string, if it isn't
mocked the 'notification's id' will be:

<MagicMock name='NotificationsRepository().create_notification()'

which can't be dump to json later.

- Bump Mako to 1.0.7 in lower-constraints.txt
- Bump decorator to 4.1.0 in lower-constraints.txt

Change-Id: I1ba563fd3144241127efe1cedf8853603dcca008
2021-09-30 00:09:22 +02:00
Martin Chacon Piza 7466a2f962 Replace removed DB methods with current ones
Replace removed Binary with LargeBinary import
Remove reflect=True from Alembic MetaData
Replace removed idle_timeout with connection_recycle_time option

- Binary was removed in SQLAlchemy 1.4.x [1]
- SQLAlchemy was updated to 1.4.15 in u-c [2]
- idle_timeout was removed in oslo.db 10.0.0 [3]
- oslo.db was updated to 10.0.0 in u-c [4]
- idle_timeout was already deprecated and renamed as
  connection_recycle_time [5]

[1] https://github.com/sqlalchemy/sqlalchemy/issues/6263#issuecomment-819645247
[2] dc86260b28
[3] a857b83c9c
[4] f322cc13d8
[5] 6634218415

Change-Id: I13ec9c2b53174cfb2e3cb990ec773588cf68007c
2021-07-22 22:35:58 +02:00
Martin Chacon Piza a6e56ab63c Add Support for Falcon 3.0.0, 3.0.1
- Support to new Falcon 3.0.0 and 3.0.1 keeping compatibility
for version 2.0.0

- Remove Falcon's class OptionalRepresentation
Starting from Falcon 3.0.0 version the class OptionalRepresentation
was removed. [1]

- Remove unnecessary URL slashes which are not compatible
with Falcon >= 3.0.0

- Keep facon.API instead of new falcon.App to keep support for
version 2.0.0

- Disable temporary docker-build and docker-publish Zuul jobs.

[1] https://falcon.readthedocs.io/en/stable/changes/3.0.0.html#breaking-changes

Change-Id: Ifb067429dd66fd350110187ac3a8b6a9977bad90
2021-06-06 21:29:22 +02:00
Ghanshyam Mann dea6f95c15 [goal] Deprecate the JSON formatted policy file
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: Ibfb162f88cb04c0b2af3fbf41cfcd96bc7e351be
2021-02-02 14:36:06 +00:00
Martin Chacon Piza 041f295b6a Add Influxdb Python client v5.3.1 compatibility
- Add a checker of version using the method ping()
if it failes, maybe because of an old version,
the flow will try with SHOW DIAGNOSTICS.

- Keep the timestamp output to 3 decimal digits as
it is working currently, independently of the version
of Influx Python Client (v5.2.3 or v5.3.1).

- Remove the support for Influxdb (the database)
older than v0.11.0

- Unittests: Adding data to handle more than 3 decimal
digits in timestamp, updating the tests to use mocks
for Influxdb from_0.11.0, creating the mocks with
from_0.11.0 explicitly.

- This change fixes monasca-tempest-python3-influxdb
Zuul job.

Change-Id: I5f8e6d2f0b56813f54fe025f91996b9d6863eadc
Story: 2007624
Task: 39658
2020-12-02 22:03:16 +01:00
Martin Chacon Piza 43922f8223 [goal] Migrate testing to ubuntu focal
As per victoria cycle testing runtime and community goal[1]
we need to migrate upstream CI/CD to Ubuntu Focal(20.04).

-Bump the lower constraints for required deps which added python3.8 support
in their later version.

-Changing the way to install and configure Zookeeper.
Installing Zookeeper from official Apache's tarball.
Adding the possiblity to set the specific Zookeeper version.
Minor change in zookeeper logger.

-Use mariadb JDBC for monasca-thresh in devstack, since Drizzle isn't
compatible with MySql Server v8.0.x which is default in Focal

-Python 3.8 doesn't seem to like dictionary keys changing during
iteration.

Fixing RuntimeError: dictionary keys changed during iteration.

Tech. details:
It runs well in py27: 5 iterations
It runs risky in py37: 7 iterations
It is forbbiden in py38: raised RuntimeError
Fixed with list(dic.items()) or tuple(dic.items())

dic = {'1': 'a', '2': 'b', '3': 'c', '4': 'd', '5': 'e'}
for key, value in dic.items():
    print("Key: {0} Value: {1}".format(key,value))
    del dic[key]
    print(dic)
    dic[key] = value
print(dic)

Story: #2007865
Task: #40197
Depends-On: https://review.opendev.org/756859
Change-Id: Ieb4cf38038ffb4d1a152f8ab3b64a14098c7cbb3
2020-10-12 15:38:05 +02:00
kuangcx 721d5e8fb5 make the variable name more detailed
Change-Id: I67961eca692e4fe79fd74ba16f3cf25edb7ee3ea
2020-08-05 16:33:18 +08:00
Witek Bedyk 3014c840d6 Set legacy_kafka_client_enabled = False on default
We change the default value of kafka.legacy_kafka_client_enabled from
True to False. The use of new Confluent Kafka client is recommended.

DevStack plugin does not set this option anymore.

Depends-On: https://review.opendev.org/740959
Depends-On: https://review.opendev.org/740966
Change-Id: I4d57b8893a6a131769009dc3299789d3fc89bab6
Story: 2007924
Task: 40338
2020-07-14 16:15:23 +02:00
Hervé Beraud 9b79bf633b Use unittest.mock instead of mock
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.

Change-Id: I4c7cb63e0a816b361c2544b1be34d8a6dadeb5c0
2020-06-09 01:23:56 +02:00
Hervé Beraud 72bc9a91ab Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: Ifd61bc8e043ac9af446ae31bffd82f67f34facbc
2020-06-02 20:24:41 +02:00
Zuul 2d8ae05899 Merge "Use unittest.mock instead of third party mock" 2020-04-23 12:30:09 +00:00
Zuul c7e0ecdfc5 Merge "Update hacking for Python3" 2020-04-23 11:08:18 +00:00
Sean McGinnis 3182d90bbf Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: Ib2843c62ff29b269139981f067ae6afcab624799
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-04-23 12:21:44 +02:00
Zuul 176985da7c Merge "Allow users to set periodic notifications on all notification types" 2020-04-22 10:09:43 +00:00
Andreas Jaeger abad342eb2 Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found by updated hacking version.

Remove hacking and friends from lower-constraints, they are not needed
there at all.

Change-Id: I35d848e9af297d3561ea2838a4808166d1c36601
2020-04-22 07:37:55 +00:00
Martin Chacon Piza 0b7dc58296 Migrate from ujson to simplejson
The change updates the imports to use simplejson library and
monasca_api.common.rest instead of monasca_common.rest, since
it was moved to this project during the API's merge.

Temporarily set following jobs as non-voting:

* monasca-tempest-python3-influxdb
* build-monasca-docker-image
* publish-monasca-api-docker-image

Change-Id: Ife3d2c9795a9dc406c2927cc9a077dda01c183c6
Story: 2007549
Task: 39389
2020-04-18 10:28:24 +02:00
Witek Bedyk 394be3f330 Use Confluent Kafka producer for logs
As already implemented for metrics, also logs should be published to
Apache Kafka using the new Kafka client.

Change-Id: Ie909b13c7692267e787d481f4de658db3b07a1c4
Story: 2003705
Task: 36866
2020-01-27 08:20:40 +00:00
Doug Szumski 92fbb93091 Allow users to set periodic notifications on all notification types
In some cases, users may want to send periodic notifications for
notification types other than webhooks.

Story: 2006837
Task: 37417
Depends-On: https://review.opendev.org/#/c/694596
Change-Id: Ia2c50e623aa79e06d2d35df4735fb2805fbf40ed
2020-01-21 10:49:22 +00:00
Doug Szumski 712f693a72 Fix notification method type DB schema migration
The Stein release does away with the concept of built in notifications,
in favour of treating all notification types equally.

This patch fixes an issue with the DB schema migration associated
with this change, which will fail if any notifications using
built-in notification types are configured at the time of the upgrade.

Story: 2006984
Task: 37746
Change-Id: I2e3f08edf1ab6aec526ad93d04effb91ddca600a
2020-01-14 09:27:30 +00:00
Witek Bedyk a76a745978 Set maximum buffer size for Kafka producer
The change sets queue.buffering.max.messages configuration option for
Kafka producer effectively limiting the number of messages in the buffer
before sending them to Apache Kafka.

Depends-On: https://review.opendev.org/694738
Change-Id: I6ebd4e21e9d55d1ac836e92dd8bf02a678170c68
Story: 2006059
Task: 37532
2019-11-18 17:57:24 +01:00
Doug Szumski f4dce6c37d Support batching metrics when writing to Kafka
When a large post (> 10s of MB) is made to the Monasca API an attempt
is made to write these metrics to the metrics topic in Kafka. However, due to
the large size of the write, this can fail with a number of obscure errors
which depend on exactly how much data is written. This change supports
splitting the post into chunks so that they can be written to Kafka in
sequence. A default has been chosen so that the maximum write to Kafka
should be comfortably under 1MB.

A future extension could support splitting the post by size, rather than the
number of measurements. A better time to look at this may be after the
Python Kafka library has been upgraded.

Story: 2006059
Task: 34772
Change-Id: I588a9bc0a19cd02ebfb8c0c1742896f208941396
2019-11-15 16:33:15 +00:00
Zuul 38bd4ce6ea Merge "Fix for editing an Alarm Definition" 2019-11-12 10:32:45 +00:00
Martin Chacon Piza 12dee6996c Fix for editing an Alarm Definition
This change solves the issue when: after removing the deterministic option
of an alarm definition by editing it, the alarm(s) asociated continue(s)
to behave as deterministic. Please see the story for more info.

Added Unittest.

Change-Id: I7743f2d2b8cd7c83541f77c7821f9512fb8abc36
story: 2006750
task: 37233
2019-11-05 13:56:06 +01:00
Ghanshyam Mann 159bc64f41 Correct the IPv6 address and port parsing
There can be lot of possibility for IPv6 address with port,
for example [::1]:80 or [2001:db8:85a3::8a2e:370]:7334.

Parsing that in more standard way is provided by oslo_uilts.netutils
parse_host_port() method[1].

Also unquote '[]' the SERVICE_HOST in case of IPv6 case so that
DB can listen on correct address.

Story: #2006309
Task: #36028

[1] 1b8bafb391/oslo_utils/netutils.py (L37)

Change-Id: I2d0ef40ab71f60564549d031185f99bc7eec40a7
2019-10-22 10:19:07 +00:00
Zuul 313a3ad7af Merge "Merge log-api and api" 2019-09-27 17:53:02 +00:00
Bharat Kunwar 967b918bae Implement separate db per tenancy
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.

This changeset implements the changes on monasca-api which handles read
requests to InfluxDB database.

It also updates the relevant docs providing link to the migration tool
which enables users to migrate their existing data to a database per
tenant model.

Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36073
2019-09-27 14:24:37 +00:00
Zuul b2c9e1551f Merge "Support time range to query dimension names/values" 2019-09-27 10:48:51 +00:00
Bharat Kunwar 233ea9c51b Support time range to query dimension names/values
At present, dimensions are not scoped by time window, which makes
dimension related queries to large databases timeout because it searches
all of time instead of a time window specified on the grafana app.

This commit implements the server side changes required to scope the
search query by the time window specified on the app.

Change-Id: Ia760c6789ac0063b8a25e52c9e0c3cc3b790ad2d
Story: 2005204
Task: 35790
2019-09-26 15:11:19 +00:00
Adrian Czarnecki d2379a018e Merge log-api and api
*Merge monasca-log-api into merge-api
*Enable logs endpoints
*Add configuration that allow to enable/disable metrics and logs part
*Remove redundant log-api source code

Story: 2003881
Task: 30533
Change-Id: Iaa5689694e7081f3375f1a2235cad31d6a7b5f76
2019-09-26 12:02:20 +02:00
Witek Bedyk 47c5ad37d5 Use Confluent Kafka client
The change introduces the possibility to run the API with the new
confluent-kafka client. It has to be enabled in the configuration file.

Story: 2003705
Task: 35859

Depends-On: https://review.opendev.org/680653
Change-Id: Id513e01c60ea584548c954a8d2e61b9510eee8de
2019-09-24 09:41:17 +00:00
Zuul ec6daef9dd Merge "Merge monasca-common code into the monasca-api" 2019-08-05 15:01:23 +00:00
Martin Chacon Piza 2485e39b53 Merge monasca-common code into the monasca-api
This change copies the code from monasca-common used by the 3
monasca APIs into monasca-api for the Merge-APIs target.

After mergin the APIs the duplicated code can be removed from
monasca-common.

Change-Id: I52d36fad846637baf10516f5cbbedc541d4c2064
Story: 2003881
Task: 30427
2019-07-26 14:08:31 +02:00
Shubham82 6cc6c9ba4f Switch python3 versions of test jobs to match Train PTI
In Train, we will use python 3.6 and 3.7 for python3 runtime
in our gate jobs [1]. This commit also adds python 3.7.
In Python3.7 async is a reserved keyword so replacing it with is_async.

[1] https://governance.openstack.org/tc/reference/runtimes/train.html

Change-Id: I05f40c4a9304cad551cefd4f10c3ba9a72d69a6f
2019-07-26 14:31:42 +05:30
Zuul 142c567886 Merge "In Python3.7 async is a keyword [1]" 2019-07-15 09:32:59 +00:00
Zuul fb23e914b1 Merge "Add unit test" 2019-07-08 17:49:11 +00:00
Adrian Czarnecki 1ef1aa1acd Add unit test
*Add unit test for helpers module
*Add unit test for metrics and notification endpoints

Story: 2006177
Task: 35701
Change-Id: Ie424f032588e4ad6afea3ea599b6febf9dd7f479
2019-07-08 13:30:18 +02:00
akhiljain23 0fae94fac2 Update hacking version to 1.1.x
This commit updates hacking version in test-requirements and
fixes some related pep8 issues

Change-Id: I67d85eb5bef72c38cc5360b5625d6b1c37adb40f
Story: 2004930
Task: 29315
2019-06-20 04:52:24 +00:00
Isaac Prior 8eea21c00d Fix sqlalchemy deprecation error
Running unit tests fails with the error:
monasca_api.tests.test_a_repository.TestAlarmRepoDB.test_should_count ...FAILED
'Textual column expression 'metric_name' should be explicitly declared with
text('metric_name'), or use column('metric_name') for more specificity'
Using strings as SQL fragments was deprecated in v1.0:
(https://docs.sqlalchemy.org/en/13/changelog/migration_10.html#warnings-emitted-when-coercing-full-sql-fragments-into-text)
Deprecation warnings now throw exceptions as of v1.3:
(https://docs.sqlalchemy.org/en/13/changelog/migration_13.html#deprecation-warnings-are-emitted-for-all-deprecated-elements-new-deprecations-added)

This patch modifies the sql query to make use of column constructor.

Change-Id: Ic258a3f453e95844249eaa763aa507d6be0d132e
2019-06-14 15:45:55 +01:00
zhangjianweibj dbe1ea8b50 Support cassandra connection timeout option
Creating a cassandra connection can be limited by connection_timeout option.

Story: 2005450
Task: 30502

Change-Id: I8803e28fe8c2c11e819be44db4ef93cb19b47a1d
2019-05-30 11:34:15 +00:00
Doug Szumski a9cc4bb482 Add support for using Falcon 2.0.0
Falcon 2.0.0 introduces some breaking changes. The relevant ones here are:

- falcon.testing.TestCase.api property was removed
- falcon.testing.TestBase class was removed

Additionally, the default behaviour for handling trailing slashes on
URIs also changed:

https://falcon.readthedocs.io/en/latest/user/faq.html#how-does-falcon-
handle-a-trailing-slash-in-the-request-path

This commit adds support for using the new release. It currently makes
no effort to be backwards compatible with older releases.

The change also updates the requirements for influxdb and sphinx
libraries to match global requirements.

Until monasca-log-api implementation is not updated to support the new
version of Falcon, `monascalog-python3-tempest` is marked to be
non-voting as agreed in the team meeting.

Story: 2005695
Task: 31015
Change-Id: I03bc8d502a333a7a71d9c12b8ddc7c5dc0a4f588
2019-05-29 16:10:53 +02:00
zhangjianweibj 0b1bb93206 cassandra cluster no load-balancing policy
monasca-api connect to cassandra cluster without
load-balancing policy parameter.
Task: 29958
Story: 2005196

Change-Id: Ia740f20f6b8d557a74a48cd0c78cd07c1500ca94
2019-04-11 09:33:18 +08:00
Zuul 6bfb250594 Merge "Update /v2/alarms/count api endpoint" 2019-04-08 16:40:05 +00:00
Ethan Apodaca 1d3efdb215
Update /v2/alarms/count api endpoint
* Brings alarms count endpoint to parity with the alarms list endpoint
* Brings alarms count endpoint to parity with the alarms counnt endpoint
  in the depricated java api
* Allow metric_dimensions filter to filter on multiple dimension values:
  metric_dimensions=dns|compute|nova

Change-Id: I46ca0e6a6da46cb850af44768de237e41a43484a
Story: 2005311
Task: 30216
2019-04-03 10:06:06 -07:00
Joseph Davis d6852ee1b7 Detect missing metric_id before passing through bytearray()
It is possible for a row in Cassandra to have a missing metric_id
(shows as 'null' in cqlsh).  This causes an ugly NoneType error
to be passed up to the user on the command line for
'monaca metric-list'.

Fix is to detect the missing value, log an error, and return the
row with None for the metric_id.

Change-Id: Ie617932c6b12a6cfe441510e120bb77a3470b9cf
Story: 2005305
Task: 30194
2019-03-26 14:17:18 -07:00
zhangjianweibj 1f0df6a985 Configure Cassandra cluster port
monasca-api can not config cassandra port.
Task: 29872
Story: 2005156

Change-Id: I007cc5f76d401b82cf5dc3c8819ec2bb3faf3bca
2019-03-14 18:47:21 +08:00
Adrian Czarnecki e18ed3f02d Improve tests coverage
* Add unit tests
* Fix python3 and python2 compatibility

Story: 2003881
Task: 29254

Change-Id: Ib54b94737dace3104976321e38ad1a9076a46877
2019-03-11 14:49:33 +00:00
Adrian Czarnecki 55a172c072 Remove redundant code
It is impossible to execute this code because
old_sub_alarm_defs_by_id value is always empty dict.

Change-Id: Id0ae84c4bc96a18185db1e825cd11c7d2e88d2b1
2019-03-05 10:19:41 +00:00