Commit Graph

175 Commits

Author SHA1 Message Date
Hasan Acar a24fd834d0 fix: migrate CI to jammy
fix: change tox.ini

fix: change queries for list_dimension_names and list_dimension_values because of influxdb time filter problem

fix: remove build_sphinx group from setup.cfg

fix: handle hashlib security problem

Change-Id: I0d31a8db5ed71c70e7b878ce5e7940e041d0fa43

Change-Id: I6f7066da10e834550cbf0c053c7bf425ac0ead93

Change-Id: If9575aee73d600bbc84fcdf58deb1c57b508d9c2

Change-Id: If515eaeee7539da3ca49997e88785dc65572b334
2024-02-22 14:35:03 +00:00
Martin Chacon Piza 7466a2f962 Replace removed DB methods with current ones
Replace removed Binary with LargeBinary import
Remove reflect=True from Alembic MetaData
Replace removed idle_timeout with connection_recycle_time option

- Binary was removed in SQLAlchemy 1.4.x [1]
- SQLAlchemy was updated to 1.4.15 in u-c [2]
- idle_timeout was removed in oslo.db 10.0.0 [3]
- oslo.db was updated to 10.0.0 in u-c [4]
- idle_timeout was already deprecated and renamed as
  connection_recycle_time [5]

[1] https://github.com/sqlalchemy/sqlalchemy/issues/6263#issuecomment-819645247
[2] dc86260b28
[3] a857b83c9c
[4] f322cc13d8
[5] 6634218415

Change-Id: I13ec9c2b53174cfb2e3cb990ec773588cf68007c
2021-07-22 22:35:58 +02:00
Ghanshyam Mann dea6f95c15 [goal] Deprecate the JSON formatted policy file
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc and tests.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: Ibfb162f88cb04c0b2af3fbf41cfcd96bc7e351be
2021-02-02 14:36:06 +00:00
Martin Chacon Piza 041f295b6a Add Influxdb Python client v5.3.1 compatibility
- Add a checker of version using the method ping()
if it failes, maybe because of an old version,
the flow will try with SHOW DIAGNOSTICS.

- Keep the timestamp output to 3 decimal digits as
it is working currently, independently of the version
of Influx Python Client (v5.2.3 or v5.3.1).

- Remove the support for Influxdb (the database)
older than v0.11.0

- Unittests: Adding data to handle more than 3 decimal
digits in timestamp, updating the tests to use mocks
for Influxdb from_0.11.0, creating the mocks with
from_0.11.0 explicitly.

- This change fixes monasca-tempest-python3-influxdb
Zuul job.

Change-Id: I5f8e6d2f0b56813f54fe025f91996b9d6863eadc
Story: 2007624
Task: 39658
2020-12-02 22:03:16 +01:00
Hervé Beraud 72bc9a91ab Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: Ifd61bc8e043ac9af446ae31bffd82f67f34facbc
2020-06-02 20:24:41 +02:00
Martin Chacon Piza 0b7dc58296 Migrate from ujson to simplejson
The change updates the imports to use simplejson library and
monasca_api.common.rest instead of monasca_common.rest, since
it was moved to this project during the API's merge.

Temporarily set following jobs as non-voting:

* monasca-tempest-python3-influxdb
* build-monasca-docker-image
* publish-monasca-api-docker-image

Change-Id: Ife3d2c9795a9dc406c2927cc9a077dda01c183c6
Story: 2007549
Task: 39389
2020-04-18 10:28:24 +02:00
Witek Bedyk a76a745978 Set maximum buffer size for Kafka producer
The change sets queue.buffering.max.messages configuration option for
Kafka producer effectively limiting the number of messages in the buffer
before sending them to Apache Kafka.

Depends-On: https://review.opendev.org/694738
Change-Id: I6ebd4e21e9d55d1ac836e92dd8bf02a678170c68
Story: 2006059
Task: 37532
2019-11-18 17:57:24 +01:00
Martin Chacon Piza 12dee6996c Fix for editing an Alarm Definition
This change solves the issue when: after removing the deterministic option
of an alarm definition by editing it, the alarm(s) asociated continue(s)
to behave as deterministic. Please see the story for more info.

Added Unittest.

Change-Id: I7743f2d2b8cd7c83541f77c7821f9512fb8abc36
story: 2006750
task: 37233
2019-11-05 13:56:06 +01:00
Bharat Kunwar 967b918bae Implement separate db per tenancy
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.

This changeset implements the changes on monasca-api which handles read
requests to InfluxDB database.

It also updates the relevant docs providing link to the migration tool
which enables users to migrate their existing data to a database per
tenant model.

Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36073
2019-09-27 14:24:37 +00:00
Zuul b2c9e1551f Merge "Support time range to query dimension names/values" 2019-09-27 10:48:51 +00:00
Bharat Kunwar 233ea9c51b Support time range to query dimension names/values
At present, dimensions are not scoped by time window, which makes
dimension related queries to large databases timeout because it searches
all of time instead of a time window specified on the grafana app.

This commit implements the server side changes required to scope the
search query by the time window specified on the app.

Change-Id: Ia760c6789ac0063b8a25e52c9e0c3cc3b790ad2d
Story: 2005204
Task: 35790
2019-09-26 15:11:19 +00:00
Witek Bedyk 47c5ad37d5 Use Confluent Kafka client
The change introduces the possibility to run the API with the new
confluent-kafka client. It has to be enabled in the configuration file.

Story: 2003705
Task: 35859

Depends-On: https://review.opendev.org/680653
Change-Id: Id513e01c60ea584548c954a8d2e61b9510eee8de
2019-09-24 09:41:17 +00:00
Zuul ec6daef9dd Merge "Merge monasca-common code into the monasca-api" 2019-08-05 15:01:23 +00:00
Martin Chacon Piza 2485e39b53 Merge monasca-common code into the monasca-api
This change copies the code from monasca-common used by the 3
monasca APIs into monasca-api for the Merge-APIs target.

After mergin the APIs the duplicated code can be removed from
monasca-common.

Change-Id: I52d36fad846637baf10516f5cbbedc541d4c2064
Story: 2003881
Task: 30427
2019-07-26 14:08:31 +02:00
Shubham82 6cc6c9ba4f Switch python3 versions of test jobs to match Train PTI
In Train, we will use python 3.6 and 3.7 for python3 runtime
in our gate jobs [1]. This commit also adds python 3.7.
In Python3.7 async is a reserved keyword so replacing it with is_async.

[1] https://governance.openstack.org/tc/reference/runtimes/train.html

Change-Id: I05f40c4a9304cad551cefd4f10c3ba9a72d69a6f
2019-07-26 14:31:42 +05:30
Zuul 142c567886 Merge "In Python3.7 async is a keyword [1]" 2019-07-15 09:32:59 +00:00
akhiljain23 0fae94fac2 Update hacking version to 1.1.x
This commit updates hacking version in test-requirements and
fixes some related pep8 issues

Change-Id: I67d85eb5bef72c38cc5360b5625d6b1c37adb40f
Story: 2004930
Task: 29315
2019-06-20 04:52:24 +00:00
Isaac Prior 8eea21c00d Fix sqlalchemy deprecation error
Running unit tests fails with the error:
monasca_api.tests.test_a_repository.TestAlarmRepoDB.test_should_count ...FAILED
'Textual column expression 'metric_name' should be explicitly declared with
text('metric_name'), or use column('metric_name') for more specificity'
Using strings as SQL fragments was deprecated in v1.0:
(https://docs.sqlalchemy.org/en/13/changelog/migration_10.html#warnings-emitted-when-coercing-full-sql-fragments-into-text)
Deprecation warnings now throw exceptions as of v1.3:
(https://docs.sqlalchemy.org/en/13/changelog/migration_13.html#deprecation-warnings-are-emitted-for-all-deprecated-elements-new-deprecations-added)

This patch modifies the sql query to make use of column constructor.

Change-Id: Ic258a3f453e95844249eaa763aa507d6be0d132e
2019-06-14 15:45:55 +01:00
zhangjianweibj dbe1ea8b50 Support cassandra connection timeout option
Creating a cassandra connection can be limited by connection_timeout option.

Story: 2005450
Task: 30502

Change-Id: I8803e28fe8c2c11e819be44db4ef93cb19b47a1d
2019-05-30 11:34:15 +00:00
zhangjianweibj 0b1bb93206 cassandra cluster no load-balancing policy
monasca-api connect to cassandra cluster without
load-balancing policy parameter.
Task: 29958
Story: 2005196

Change-Id: Ia740f20f6b8d557a74a48cd0c78cd07c1500ca94
2019-04-11 09:33:18 +08:00
Zuul 6bfb250594 Merge "Update /v2/alarms/count api endpoint" 2019-04-08 16:40:05 +00:00
Ethan Apodaca 1d3efdb215
Update /v2/alarms/count api endpoint
* Brings alarms count endpoint to parity with the alarms list endpoint
* Brings alarms count endpoint to parity with the alarms counnt endpoint
  in the depricated java api
* Allow metric_dimensions filter to filter on multiple dimension values:
  metric_dimensions=dns|compute|nova

Change-Id: I46ca0e6a6da46cb850af44768de237e41a43484a
Story: 2005311
Task: 30216
2019-04-03 10:06:06 -07:00
Joseph Davis d6852ee1b7 Detect missing metric_id before passing through bytearray()
It is possible for a row in Cassandra to have a missing metric_id
(shows as 'null' in cqlsh).  This causes an ugly NoneType error
to be passed up to the user on the command line for
'monaca metric-list'.

Fix is to detect the missing value, log an error, and return the
row with None for the metric_id.

Change-Id: Ie617932c6b12a6cfe441510e120bb77a3470b9cf
Story: 2005305
Task: 30194
2019-03-26 14:17:18 -07:00
zhangjianweibj 1f0df6a985 Configure Cassandra cluster port
monasca-api can not config cassandra port.
Task: 29872
Story: 2005156

Change-Id: I007cc5f76d401b82cf5dc3c8819ec2bb3faf3bca
2019-03-14 18:47:21 +08:00
Adrian Czarnecki e18ed3f02d Improve tests coverage
* Add unit tests
* Fix python3 and python2 compatibility

Story: 2003881
Task: 29254

Change-Id: Ib54b94737dace3104976321e38ad1a9076a46877
2019-03-11 14:49:33 +00:00
Adrian Czarnecki 55a172c072 Remove redundant code
It is impossible to execute this code because
old_sub_alarm_defs_by_id value is always empty dict.

Change-Id: Id0ae84c4bc96a18185db1e825cd11c7d2e88d2b1
2019-03-05 10:19:41 +00:00
ZhongShengping 26f5cd6b4a In Python3.7 async is a keyword [1]
Change all instances of "async" to "async_"

[1] https://docs.python.org/3/whatsnew/3.7.html

Change-Id: Ib4121d72e97da2f35adb83b257fa8775370e21df
2019-02-18 17:06:28 +08:00
Adrian Czarnecki 67513278a9 Remove unused exception
Change-Id: I3d05844db8cf268486434388e81dba8716130940
2019-01-14 12:53:28 +01:00
Vu Cong Tuan 93d012bd0f Replace usage of get_legacy_facade() with get_engine()
We've been seeing warnings emitted to the python 3 unit tests:

  OsloDBDeprecationWarning: EngineFacade is deprecated; please use
  oslo_db.sqlalchemy.enginefacade

which stem from our use of the get_legacy_facade() oslo.db method.

This replaces the get_legacy_facade() usage with the get_engine()
method from the transaction context manager.

The same has been done for Nova:
https://review.openstack.org/#/c/606213

Change-Id: I83394d79280c3e22469551f17356d057a26230c4
2018-11-06 14:36:04 +07:00
Adrian Czarnecki 7ba3a8b7ab Python3: Fix failing tempest tests
Fix sorting and string encoding

Story: 2000975
Task: 24394

Change-Id: Iaf4575b47fdc299c4e9eb77e19c36bebb0642ec8
2018-08-24 08:33:18 +00:00
Witold Bedyk 0d218ddae5 Remove usage of future library
*future* library is not listed in global-requirements and should not be
used. This commit removes all its occurences and replaces with
oslo_utils.encodeutils module.

Change-Id: I5d0795a1a894c103422f7d63f5d76b29ea6d15a4
Story: 2003193
Task: 23360
2018-08-01 14:32:33 +02:00
Zuul a0381710f2 Merge "Python 3.5: Fix alarms repository" 2018-07-03 13:05:59 +00:00
Zuul aab94c6a7e Merge "Python 3.5: Fix alarm definitions repository" 2018-07-03 13:05:58 +00:00
Zuul cc347fdd02 Merge "Alembic migrations for all SQL script revisions" 2018-07-02 21:16:06 +00:00
Zuul cb7f226a71 Merge "Add py3 support for repositories packages" 2018-06-28 10:17:57 +00:00
Adrian Czarnecki 6af07fb0fc Add py3 support for repositories packages
Story: 2000975
Task: 12579

Change-Id: Iaf01fcf9e600d1f89eb202f6a908d6de50c9578f
2018-06-27 13:29:36 +02:00
Adrian Czarnecki dd99ad56a2 Python 3.5: Fix alarms repository
Fix python3 unit test for alarms reposiotry.

Change-Id: Ibb8cba96f62ac5580cd7b74bf6f2751efa66cd90
2018-06-22 09:57:28 +00:00
Johannes Grassler 39d5bbbb0a Alembic migrations for all SQL script revisions
This commit adds

* An alembic environment for running database migrations
* Alembic database migrations for all revisions
  of the legacy SQL script currently being used to set up the
  Monasca database.
* The get_all_metadata() function for exposing full data model
  metadata. This function is needed generating migrations from
  the current state of the model and should be kept up to date
  as the model is being extended.

Change-Id: I41b4577d8334791f712177348cadfe9b07b62ef1
Story: 2001654
Task: 14339
2018-06-22 08:16:35 +00:00
Adrian Czarnecki b7c9b7ae6d Python 3.5: Fix alarm definitions repository
Change-Id: I98789e858470eb9ff77413218dc51d76c210be3a
2018-06-21 10:49:18 +00:00
Zuul 5bd1b5e212 Merge "Add py3.5 support for alarms endpoint/unit test" 2018-05-24 11:31:31 +00:00
Adrian Czarnecki 8efd03aab8 Add py3.5 support for alarms endpoint/unit test
* Python 3 dict.values() returns iterator, must be converted to list
* use `u` prefix for unicode literals
* use builtins.str for handling unicode
* use oslo_serialisation for loading JSON objects

Can be tested with:
  tox -e py35 -- -r monasca_api.tests.test_alarms.*

Story: 2001400
Task: 12580

Change-Id: I0cc7018f7d5efedb4937922b6daa2fb59b3d645c
2018-04-12 17:34:17 +02:00
Jui Chandwaskar 03bd329dd0 Update pep8 checks
* Update max line length to 100
* Clean up code for pep8 checks

Change-Id: Ibde2939831e6eeaab3db66fbc18255c71592bb41
Signed-off-by: Jui Chandwaskar <jchandwaskar@op5.com>
2018-04-11 12:40:27 +02:00
bandorf 404fe5e932 Fix:No alarms created if metric name in alarm def. expr. is mix case
Change-Id: I38cb00fe7b10bbe6732022e8fffcecd7e5eab6b5
Story: 2001539
Task: 6349
Depends-On: I739612e96a2975bde12aea91f3f8dfcd6fb15342
2018-02-20 14:44:09 +00:00
Zuul 5ba8133ad6 Merge "Provide id for existing sub-expressions of alarm definitions" 2018-02-09 17:19:06 +00:00
bandorf c1d691aeb8 Provide id for existing sub-expressions of alarm definitions
Story: 2001525
Task: 6328

Change-Id: I07916bc1ccd3691b16342db4b6c9710beecefdae
2018-02-07 17:25:19 +01:00
Zuul e67b88acce Merge "Revert "Adds tags for inhibit and silence"" 2018-02-05 12:17:26 +00:00
Zuul ad4d9df3fe Merge "Add password authentification in monasca-api with Cassandra" 2018-02-01 11:50:32 +00:00
Andrea Adams f69cb3152a Revert "Adds tags for inhibit and silence"
This reverts commit 8781a256f0.
Since this only implements part of the silence / group / inhibit
feature, reverting until the rest is ready to go

Change-Id: Ied34507feb5d3580960c918254f905ab90d9691e
2018-01-31 09:56:50 -07:00
Zuul dbb0fcb67c Merge "Statistics api failure when end time is not used" 2018-01-30 21:07:26 +00:00
James Gu 4a6015ad65 Add password authentification in monasca-api with Cassandra
Add the support to configure user and password in monasca api and enforce
the authetification when connecting to cassandra db when the user and
password is set.

Change-Id: I9e6689e0e8e6e97d5cff3a59aecb53483ec320bd
story: 2001471
task: 6191
2018-01-29 15:38:16 -08:00