Commit Graph

132 Commits

Author SHA1 Message Date
Hasan Acar 84f03c516a Fixing the gates
fix: sha256 instead of sha1 for metric_id

fix: allowlist added to .tox file
Change-Id: I0384f8e9cdae901df9403c442fce4cf1bf036968
2024-03-20 05:52:08 +00:00
Mirek Malinowski 87938f738e Add SSL support for InfluxDB connection.
Added ssl and verify_ssl options to influxdb config

Task: 44888
Story: 2009950
Change-Id: Icfe06233a178a5383441db6d87d46b93b6a63757
2022-03-30 19:52:50 +01:00
Hervé Beraud 37411e792a Add doc/requirements
We need to specify doc requirements in doc/requirements.txt
to avoid problems with the pip resolver [1] for the release team [2][3].

Removing specific doc requirements from test-requirements.txt.

The problem here is that this repos haven't doc/requirements.txt file
and by default in this case zuul will use the test-requirements.txt file
to pull requirements [4].

This requirements file contains extra requirements like flake8 that
collided with those allowed in our job environment and so the new pip
resolver fails to install these requirements and the job exits in error.

This project meet the conditions leading to the bug.

/!\/!\/!\
Notice that I voluntarily added the doc directory even if no docs
are generated here because zuul will try to pull this requirements from
there first and the contained requirements are needed for reno but AFAIK
the releasenotes dir is ignored by zuul. c.f [4] for further details.
/!\/!\/!\

Bump a series of lower-constraints and requirements to work with new pip
resolver, testing with steps outlined at [5]

Fix Flake8 E741 [6]

[1] http://lists.openstack.org/pipermail/release-job-failures/2021-January/001500.html
[2] http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019611.html
[3] http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019612.html
[4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-sphinx/tasks/main.yaml#L36
[5] http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019285.html
[6] https://www.flake8rules.com/rules/E741.html

Change-Id: Ic1504a18a780b0f517c5aa3dd3bfb04998d42e74
2021-01-18 10:55:41 +01:00
Adrian Czarnecki df0fd4958f Remove six
Change-Id: I898c97fda4e1e93c64c6b75fbfea5af80ffd684a
Story: 2008305
Task: 41274
2020-11-24 15:10:57 +01:00
Zuul bdd513131e Merge "Remove unnecessary test to bring compatibility with py38" 2020-09-01 14:49:54 +00:00
Martin Chacon Piza 0198487062 Remove unnecessary test to bring compatibility with py38
Change-Id: If58a9a60a6428ad362789f354f8f4a750210b9a8
2020-09-01 11:29:29 +02:00
Witek Bedyk 709318c3bb Set legacy_kafka_client_enabled = False on default
We change the default value of kafka_common.legacy_kafka_client_enabled
from True to False. The use of new Confluent Kafka client is
recommended.

Change-Id: I300958e1e945e5236b01f4e50cb01c18468cc524
Story: 2007924
Task: 40340
2020-07-14 15:51:31 +02:00
Hervé Beraud ac9d9b7351 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: I937b48769403d0c78af5069e29768bb7bcdb7b0b
2020-06-02 20:25:37 +02:00
Andreas Jaeger b306576d27 Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found.

Update local hacking checks for new flake8.

Remove hacking and friends from lower-constraints, it's not needed
there.

Change-Id: Iadf750a1e6d3181b645ceccdf75cf910bc23adfd
2020-04-21 19:27:10 +02:00
Witek Bedyk dc8956d7b8 Migrate from ujson to simplejson
The change updates the imports to use simplejson library and fixes three
unit tests. In particular TestUtils.test_parse_alarm_state_hist_message
has been slightly changed to make testing more robust.

Also, Python 2 specific implemetation has been removed from
influxdb.line_utils .

Additionally, standard library unittest.mock is used instead of the
third party mock lib.

Depends-On: https://review.opendev.org/718014
Depends-On: https://review.opendev.org/720188
Change-Id: I64b1a60e8be929c25c005a0f429b1274cb8570e6
Story: 2007549
Task: 39390
2020-04-21 12:35:08 +02:00
Arseni Lipinski 26844c1dd0 Add tests for cassandra/retry_policy.py
Story: 2007280
Task: 38724

Depends-On: https://review.opendev.org/#/c/679076/

Change-Id: I440e36ad37dc94ffcd665f3925b7f25b5964cbf8
2020-02-13 10:39:05 +00:00
Witek Bedyk 0696cf54ee Add configuration option influxdb.batch_size
The new configuration option is used to control the maximum size of
batches written to InfluxDB and optimizes performance for high loads.
The default value is set to 10 000 data points per batch.

Change-Id: I82c3e4e64984a996e41fe2657ce36b032b40d3fd
Story: 2007191
Task: 38319
2020-01-21 11:07:22 +01:00
Bharat Kunwar 97aab2f004 Support default_retention_hours for influxdb
At the moment, all data is retained infinitely. This change allows users
to specify a default_retention_hours (defaults to 0 meaning unlimited
retention) in the influxdb section of monasca persister configuration so
that new projects are automatically assigned this retention policy when
the given hour is greater than 0 to preserve original behaviour.

Story: 2006331
Task: 37234

Change-Id: I4136df1d43954eb026a104f3f85b3a58197f5435
2019-11-15 13:47:47 +00:00
Bharat Kunwar df12bd8628 Support automatic db creation for InfluxDB
At the moment, monasca persister assumes that an InfluxDB database
already exists. With this change, a new database is created when it does
not exist.

Story: 2006331
Task: 37327

Change-Id: I3d88eb8b2cdbd81f9c84fe1703b2ee2d1b09e1ab
2019-10-31 23:12:00 +00:00
Witek Bedyk e883b5bcda Move data_points to repository class
data_points_class can be now declared as a List or a Dictionary. The
latter is used when writing measurements per tenant. This refactoring
avoids creating the dictionary and then chaining again its values in
case of ElasticSearch and Cassandra databases or if db_per_tenant option
is disabled. The change also fixes the events pipeline.

Change-Id: I25ea80eea714acb167e70f188ed229cc90531596
Story: 2006331
Task: 37211
2019-10-24 13:34:30 +02:00
Bharat Kunwar 0daa8ebc76 Tool to migrate existing data to db per tenant
At present, all time series are accumulated in the same database in
InfluxDB. Separate database per tenant would make queries faster for
tenants. It would also allow administrators to define retention policy
per tenancy.

This changeset adds a migration tool and a guide to allow users to
migrate existing data to a database per tenant.

Task: 36435
Story: 2006331

Change-Id: I3faf3d96b1121c8b8c383dee2fda27d34af5c853
2019-09-27 08:31:16 +00:00
Bharat Kunwar 774e981f87 Implement separate db per tenancy
At present, all time series are accumulated in the same database in
InfluxDB. This makes queries slow for tenants that have less data. This
patch enables the option to use separate database per tenancy.

This task implements the changes on monasca-persister which handles
write requests to InfluxDB database.

Change-Id: I7de6e0faf069b889d3953583b29a876c3d82c62c
Story: 2006331
Task: 36072
2019-09-27 08:31:07 +00:00
zhangjianweibj c78d5ebc43 Fix InfluxdbClientError exception bug
InfluxdbClientError has content attribute no message attribute.
Message attribute deprecated in Python 2.6 and it's go away
in Python 3.0

Change-Id: I5fe4bcb018a7c8c99df29d9798aa0e2736f33b6f
Story: 2005900
Task: 33762
2019-08-21 17:08:54 +08:00
Witek Bedyk b88084003f Use Confluent Kafka client
The change introduces the possibility to run the persister with
the new confluent-kafka client. It has to be enabled in the
configuration file.

Story: 2003705
Task: 30117

Depends-On: https://review.opendev.org/675297
Change-Id: I05428b8ae9e0ba9af5b81d3b103434ebd5657108
2019-08-15 14:47:51 +02:00
Zuul 4e8b1fae05 Merge "Add tests for influxdb/metrics_repository.py" 2019-08-07 12:41:41 +00:00
Zuul a840cb413d Merge "Add tests for cassandra/token_range_query_manager.py" 2019-07-12 08:56:40 +00:00
Zuul 52b864eb5a Merge "Add tests for cassandra/alarm_state_history_repository.py" 2019-07-12 08:56:39 +00:00
Arseni Lipinski f23adaadc3 Add tests for cassandra/token_range_query_manager.py
Change-Id: Ifd3d2a2604b71a315ae560510b66e6bf9fda7ee2
2019-07-05 11:54:58 +00:00
Arseni Lipinski 1ce49ce8fe Add tests for cassandra/alarm_state_history_repository.py
Change-Id: I0790b7615132728282b4a1045f18e269e7d0bdb8
2019-07-05 07:49:24 +00:00
zhangjianweibj 506717e23e cassandra metrics_id contain null value bug
metrics table in cassandra always contain row which has no metric_id.
it is casused by cassandra ttl.cassandra supports TTL column only,
so when update a row,metric_id must update as well.

Change-Id: Ia2d2e1585965dd31f9679e2a63497bf4cc68349e
Story: 2005832
Task: 33590
2019-06-10 17:18:30 +08:00
Joseph Davis c32885aed8 Clarify dependency on cachetools in recreate-metric-id tool
It was found in some installations that python-cachetools needed
to be present on the system.  This can happen if the installation
was configured for the Java mon-persister but the recreate-metric-id
tool is written in Python.  Added a bullet item to describe this
dependency and the quick workaround of installing the
python-cachetools package.

Change-Id: I25635c069a5ed21342780a28b516c74b61dae30b
Story: 2005305
Task: 30611
2019-06-06 13:39:54 -07:00
Arseni Lipinski 8f1d672f23 Add tests for influxdb/metrics_repository.py
Change-Id: Iea4d74dffb28c3315761de464840e8c4cda57c0a
2019-06-05 14:20:54 +00:00
Zuul 945fd8d04c Merge "Replace git.openstack.org URLs with opendev.org URLs" 2019-06-04 09:39:35 +00:00
Zuul 327e5ccc20 Merge "Add tests for cassandra/metric_batch.py" 2019-06-03 14:19:00 +00:00
zhulingjie 3aa33e62a3 Replace git.openstack.org URLs with opendev.org URLs
Change-Id: I66f8508ab8e29e189a16e7155caa21484486a825
2019-06-03 15:33:15 +02:00
Zuul 3841353945 Merge "Add tests for repositories/persister.py" 2019-05-30 15:09:03 +00:00
Arseni Lipinski e21e04436f Add utils.py tests
Change-Id: Id51645d0696c8a46d9c7fe81446c015e4d0addc6
2019-05-23 12:29:48 +02:00
Joseph Davis 5e1823de06 Fix a typo and add troubleshooting in tools README
It was discovered that the example paths included in the README.rst
used monasca-persister rather than monasca_persister.

Also added a footnote with two error messages that occur when the
contact_points is not specified correctly.

Story: 2005305
Task: 30611

Change-Id: I0c53af817504e638bd82401ca0b62874cb591f36
2019-05-16 15:46:39 -07:00
Zuul f37811fdef Merge "Add tests for influxdb/alarm_state_history_repository.py" 2019-05-16 08:55:46 +00:00
Zuul 19d4358893 Merge "Minor refactoring for utils.py" 2019-05-15 23:41:44 +00:00
Joseph Davis 0aaa7d357f Create a tool for recreating missing metrid_id in Cassandra
In some rare cases it is possible for a row in Cassandra for
metrics to have no value for metric_id or created_at, though
they may still have updated_at and the other required columns.

This tool is for recreating the metric_id from the other required
columns.

An additional 'persister-check-missing-metric-id.py' tool is
provided which can be run to see if there are missing metric-id
values that need to be recreated.

Please see the README.rst for usage directions.

Story: 2005305
Task: 30611

Change-Id: I0593558407c8c773d728bbd035dde91310b59be3
(cherry picked from commit 09af9bff91)
2019-05-13 09:55:49 -07:00
Arseni Lipinski f4ea2f9db0 Minor refactoring for utils.py
Change-Id: Ie40f22f768e67f112081524446174383845fb9b7
2019-05-08 13:36:28 +02:00
Arseni Lipinski 7f2da27a3a Add tests for repositories/persister.py
Change-Id: I4443a2b205199d81ab8d05d17e65a47485de8f2f
2019-05-07 16:17:19 +02:00
Arseni Lipinski 1f062298a6 Add tests for influxdb/alarm_state_history_repository.py
Change-Id: Ic92623339f8e9fa6d7bcbb5e4bd02328332020a2
2019-04-26 11:23:01 +00:00
Arseni Lipinski 54df2c58b4 Add tests for cassandra/metric_batch.py
Change-Id: I048b77a7bef7e8477081ba2a404e919897752b7b
2019-04-23 14:42:29 +02:00
Zuul eda5e3ae79 Merge "Change process_message() to static" 2019-04-18 15:40:16 +00:00
Arseni Lipinski e415ea71cb Change process_message() to static
Change-Id: I931a93f9de0df3373e5601a9d7abb29e7f90a8bc
2019-04-12 15:09:53 +02:00
Doug Szumski f43d019a67 Widen exception catch for point parse failure
This change supports catching more parse errors which can jam
the persistor, such as:
`partial write: unable to parse 'string_with_invalid_char.%'`.

Change-Id: Ia12d04708a88308e88fcbd488d72f4313ba42a36
Story: 2005430
Task: 30468
2019-04-11 10:13:39 +01:00
zhangjianweibj 2bef4e500a cassandra contact_points bug
persister get contact_points from configuration file
with "[ip.dest for ip in conf.cassandra.contact_points]".
cfg.dest replace "-" with "_".if user config cassandra
hosts cassandra-0,then persister crashed.
Task: 29957
Story: 2005195

Change-Id: I9ee29fdce781739f06683e4b4a80183669663189
2019-04-02 15:53:39 +08:00
zhangjianweibj 0ab568481f some points unable to parse
some metrics get from kafka,may not a standard point that
influxdb can parse.in this case,points should be dropped.

Task: 29641
Story: 2005069

Change-Id: Ib907a280ced26ef2c228efe14b57440f894d8aad
2019-03-19 10:58:14 +01:00
Zuul 399b9adeec Merge "Update documentation to use new default config" 2019-02-18 19:26:22 +00:00
Zuul c4822d6f8e Merge "Replace persister.conf with monasca-persister.conf" 2019-02-18 17:28:25 +00:00
Witek Bedyk 0e59d8b1ce Update documentation to use new default config
The new default configuration file name has changed to
monasca-persister.conf. This change updates documentation to use the new
default.

Change-Id: Ic271221f19cdc86e8297ed88cdcf4a86df4374b8
Story: 2004867
Task: 29534
2019-02-18 18:21:25 +01:00
Zuul 76885d6b5d Merge "Fix default alarm_state_history_driver value" 2019-02-13 16:35:40 +00:00
Thomas Bechtold a59f436e69 Replace persister.conf with monasca-persister.conf
persister.conf is the deprecated path for the configuration file.
The new path is /etc/monasca/monasca-persister.conf . This is inline
with other OpenStack projects.
Also fix the log setup and set the product name to 'monasca-persister'
which is inline with other OpenStack and Monasca projects.

Change-Id: Ida11c326b3e6771b5ccf994205c55874db05bc5c
Story: 2004867
Task: 29112
2019-02-08 11:03:31 +00:00