Commit Graph

1167 Commits

Author SHA1 Message Date
Zuul b611e60954 Merge "Update master for stable/2024.1" 2024-03-13 14:44:21 +00:00
Zuul eaadb1427e Merge "reno: Update master for unmaintained/victoria" 2024-03-13 14:36:38 +00:00
Zuul 9a6e167721 Merge "reno: Update master for unmaintained/wallaby" 2024-03-13 14:36:37 +00:00
Zuul 38b4e41b1b Merge "reno: Update master for unmaintained/xena" 2024-03-13 14:32:31 +00:00
OpenStack Release Bot f1a2273f29 Update master for stable/2024.1
Add file to the reno documentation build to show release notes for
stable/2024.1.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.1.

Sem-Ver: feature
Change-Id: Ic60212f53d02906092c8656b1fab8d9bb84d54a8
2024-03-12 11:15:15 +00:00
Matt Crees 2a0904db00 Set Elastic/OpenSearch jobs as voting in Zuul
Given these backends are now used in many production environments, they
can no longer be considered experimental.

Change-Id: I9e9f3023bf2a50807540e69b764600c0c5f995d5
2024-03-06 11:52:18 +00:00
Pedro Henrique 3114c3f979 Add grenade tests
Change-Id: I427d4b91cc5da9e9b2ffdb4239a5256ab44a1e06
2024-03-05 18:06:11 +01:00
OpenStack Release Bot 94b822f001 reno: Update master for unmaintained/xena
Update the xena release notes configuration to build from
unmaintained/xena.

Change-Id: I3f6386e6705a5cf4e03044f64347ec6b35c1d0ea
2024-03-05 14:55:07 +00:00
OpenStack Release Bot 22df075ebb reno: Update master for unmaintained/wallaby
Update the wallaby release notes configuration to build from
unmaintained/wallaby.

Change-Id: I87f39794a1d91796745d3d55f84b60ca96e39090
2024-03-05 14:54:42 +00:00
OpenStack Release Bot 1b2eb41e0f reno: Update master for unmaintained/victoria
Update the victoria release notes configuration to build from
unmaintained/victoria.

Change-Id: I37026520ed55a632cb006aafff891b26673eea5f
2024-03-05 14:54:18 +00:00
Matt Crees f2c103535e Replace usage of LegacyEngineFacade
Switch to using oslo_db.sqlalchemy.enginefacade instead, as this is
required for SQLAlchemy 2.x support.

Change-Id: Ifcad28239b6907b8ca396d348cbfa54185355f68
2024-02-28 14:59:13 +00:00
Zuul 3e35eb05ed Merge "Add requests to requirements" 2024-02-19 16:03:51 +00:00
Zuul 85ad6904fd Merge "Add support to InfluxDB v2 as storage backend" 2024-02-19 16:03:46 +00:00
Zuul 328307512b Merge "Removal of Monasca fetcher and collector" 2024-02-19 15:12:50 +00:00
Zuul 9ca4a42189 Merge "Fix devstack runprocess for cloudkitty api" 2024-02-19 15:01:16 +00:00
Pedro Henrique 101a410739 Add support to InfluxDB v2 as storage backend
This patch allows CloudKitty to use InfluxDB v2 with Flux queries. This
type of query uses less CPU and RAM to be processed in the InfluxDB
backend.

Change-Id: I8ee3c92776aa69afbede353981a5fcd65dd7d099
Depends-On: https://review.opendev.org/c/openstack/requirements/+/895629
Story: 2010863
Task: 48539
2024-02-19 15:30:37 +01:00
Pierre Riteau 9797989e88 Add requests to requirements
We use requests in several places in cloudkitty code, so we need to
depend on it directly instead of installing it through dependencies.

Change-Id: Ibedb9ddba61b39c80b1e1b3910a90468bdfc76ae
2024-02-19 09:32:01 +01:00
Rafael Weingärtner b46dcd8983 Removal of Monasca fetcher and collector
Change-Id: I314c751f2b2dea693bea66d7d9b06679a06b0b6b
2024-02-14 08:59:28 -03:00
Pedro Henrique 25a7306412 Fix devstack runprocess for cloudkitty api
After the commit
8778c64776

The module PBR in openstack started to validating the parameters when
creating an embedded WSGI, now if invalid arguments are given
to the cloudkitty-api it raises an error as we are facing in the
devstack when using `CLOUDKITTY_USE_MOD_WSGI=False`:

cloudkitty-api[86126]: usage: cloudkitty-api [-h] [--port PORT] [--host IP] -- [passed options]
cloudkitty-api[86126]: cloudkitty-api: error: unrecognized arguments: --config-file=/etc/cloudkitty/cloudkitty.conf

This PR also extracts the upgrade database workflow to a function
to be used in grenade tests

Change-Id: Ifc1a8563a9efcae2abaa6f8eb036405a93ff296d
2024-02-12 10:33:48 -03:00
OpenStack Release Bot 6414edca51 reno: Update master for unmaintained/yoga
Update the yoga release notes configuration to build from
unmaintained/yoga.

Change-Id: I889b90c6cc3e569b77ebdfc6ea1d3fe8a47288ae
2024-02-05 16:43:04 +00:00
Ghanshyam Mann 9b2d97c6d4 Update python classifier in setup.cfg
As per the current release tested runtime, we test
till python 3.11 so updating the same in python
classifier in setup.cfg

Change-Id: If66420bae76e8db527c1aa191cb31d6ed8a7ff91
2024-01-28 22:40:42 +00:00
Rafael Weingärtner 3aa31f8719 Add description option to a rating metric definition
There is the need to add human-readable description to the metric
definition. This can then be used to create custom reports in the
`summary` GET API. The value has to be stored in the backend as
we do with the alt_name and unit of the metric as well.

Depends-On: https://review.opendev.org/c/openstack/cloudkitty/+/861786
Change-Id: Icea8d00eaf3343e59f0f7b2234754f6abcb23258
2024-01-28 14:41:18 -03:00
Zuul 8eb8c4931a Merge "Add OpenSearch as a v2 storage backend" 2023-12-01 16:52:14 +00:00
Zuul bf208c517e Merge "Add groupby options by different timeframes" 2023-11-27 15:22:12 +00:00
Zuul 5ebb525820 Merge "Clean up release note" 2023-11-27 14:46:49 +00:00
Matt Crees 964c6704a2 Add OpenSearch as a v2 storage backend
To facilitate the switch from Elasticsearch to OpenSearch, the ES
backend has been duplicated and renamed where appropriate to OpenSearch.

The OpenSearch implementation was modified in places for compatibility
with OpenSearch 2.x, for example:

- remove mapping name from bulk API URL
- replace put_mapping by post_mapping

This will allow for the future removal of the Elasticsearch backend.

Change-Id: I88b0a30f66af13dad1bd75cde412d2880b4ead30
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
2023-11-27 15:19:14 +01:00
Zuul c842ea6ab2 Merge "Optimize CloudKitty reprocessing process" 2023-11-27 13:48:26 +00:00
Zuul 6075cdce95 Merge "Patch for `use_all_resource_revisions` option" 2023-11-27 13:48:25 +00:00
Rafael Weingärtner 45f5e72472 Add groupby options by different timeframes
Introduce new default groupby options: (i) time: to group data by
hourly; (ii) time-d: to group data by day of the year; (iii) time-w:
to group data by week of the year; (iv) time-m: to group data by month;
and, (v) time-y: to group data by year. If you have old data in
CloudKitty and you wish to use these group by methods, you will need
to reprocess the desired timeframe.

Story: #2009839
Task: #44438
Depends-On: https://review.opendev.org/c/x/wsme/+/893677
Change-Id: Iad296f54f6701af84e168796aec9b1033a2a8a2d
2023-11-13 11:25:03 -03:00
Zuul fe3b8f2bdc Merge "Remove the manual patch for WSME" 2023-10-16 15:47:42 +00:00
Pierre Riteau 12347e16c8 Fix retrieval of reprocessing tasks
Calling GET /v2/task/reprocesses with python-cloudkittyclient was
returning Internal Server Error, with the following API trace:

      File "/var/lib/kolla/venv/lib/python3.6/site-packages/cloudkitty/api/v2/task/reprocess.py", line 259, in get
        order, ACCEPTED_GET_REPROCESSING_REQUEST_ORDERS)
    TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given

This was because http_exceptions.BadRequest was given multiple arguments
(similar to LOG.* methods) instead of a single string.

Another issue is that python-cloudkittyclient sends the "DESC" order
while the API only supports "desc" and "asc". Convert to lower case for
compatibility.

Change-Id: Id1145adff82bc9a01e4eb0f306f0bfa535142459
2023-10-13 10:27:49 +02:00
Zuul b96b9afc68 Merge "Use correct metadata for metrics gathered from gnocchi" 2023-10-02 17:28:30 +00:00
Zuul c3fff2d464 Merge "Fix a concurrency issue when locking reprocessing tasks" 2023-10-02 15:36:36 +00:00
Rafael Weingärtner 5c2f9e7f71 Optimize CloudKitty reprocessing process
Currently, when a reprocessing task is scheduled, CloudKitty executes
the cleaning of the data for the reprocessing period in one hour
fashion (the default period). Therefore, for each one of the
timeframes, a delete query is sent to InfluxDB (when using it as a
backend). However, InfluxDB is not a very optimized time series database
for deletion; thus, this workflow generates quite some overhead and
slowness when reprocessing. If we clean right away the whole time
frame for the reprocessing task, and then we just reprocess it, it will
execute a single delete query in InfluxDB, which has a similar cost as
a delete to remove the data for a single time frame.

This patch optimized the reprocessing workflow to execute batch cleaning
of data in the storage backend of CloudKitty.

Change-Id: I8282f44ad837c71df0cb6c73776eafc7014ebedf
2023-10-02 11:50:37 -03:00
Rafael Weingärtner b796efd45f Patch for `use_all_resource_revisions` option
The option 'use_all_resource_revisions' is useful when using Gnocchi
with the patch introduced in [1]. That patch can cause queries to
return more than one entry per granularity (timespan), according to
the revisions a resource has. This can be problematic when using the
'mutate' option of Cloudkitty. Therefore, this option
('use_all_resource_revisions') allows operators to discard all
datapoints returned from Gnocchi, but the last one in the granularity
that is queried by CloudKitty. The default behavior is maintained,
which means, CloudKitty always uses all the data points returned.

However, when the 'mutate' option is not used, we need to sum all the
quantities and use this value with the latest version of the attributes
received. Otherwise, we will miss the complete accounting for the time
frame where the revision happened.

[1] https://github.com/gnocchixyz/gnocchi/pull/1059

Change-Id: I45bdaa3783ff483d49ecca70571caf529f3ccbc3
2023-10-02 11:46:03 -03:00
Rafael Weingärtner a37dda5013 Use correct metadata for metrics gathered from gnocchi
Gnocchi fixed its `aggregates` API with PR
https://github.com/gnocchixyz/gnocchi/pull/1059. Before that patch,
the `aggregates` API would only return the latest metadata for the
resource of the metric being handled. Therefore, for CloudKitty
processing and reprocessing, we would always have the possibility of
using the wrong attribute version to rate the computing resources.
With this patch we propose to always use the correct metadata for the
processing and reprocessing of CloudKitty. This means, we always use
the metadata for the timestamp that we are collecting at Gnocchi.

The patch was released under version 4.5.0 of Gnocchi.

Change-Id: I31bc2cdf620fb5c0f561dc9de8c10d7882895cce
2023-10-02 11:35:08 -03:00
Rafael Weingärtner a89aa44535 Remove the manual patch for WSME
WSME was released. Therefore, we can remove this work around.

Change-Id: I1673258da4fa9bcd96db24d635d9f06e3fc4a6a2
2023-09-21 15:45:50 -03:00
Rafael Weingärtner 0451df0d1b Fix a concurrency issue when locking reprocessing tasks
It was discovered that in some situations the same reprocessing task
might be processed simultaneously by different workers, which can
lead to unnecessary processing. This was happening due to the use
of "current_reprocess_time" in the lock name, which would lead to
different locking name for some situations; for instance, when worker
start processing a brand new reprocessing task, and after reprocessing
a few time frames, the "current_reprocess_time" is updated, then when
other workers achieve the same locking moment, they would have a
different lock name for the same scope ID, and reprocess a scope
that is currently in reprocessing.

Change-Id: I487d0eeb1cedc162d44f8c879a27f924b5c76206
2023-09-18 11:40:38 -03:00
Zuul 52b51b004d Merge "Remove cloudkitty.conf.sample from docs" 2023-09-18 13:48:46 +00:00
Pierre Riteau fe1d03ccde Clean up release note
Release notes are aimed for end users or operators, not for developers.

Change-Id: I0fb936b9c986fa159f6a19fffaad031ea6ff6df3
2023-09-18 15:48:23 +02:00
Zuul 31f825d3af Merge "Optimizing SQL queries that filter on a time range" 2023-09-18 11:57:04 +00:00
Rafael Weingärtner 5a45386477 Remove cloudkitty.conf.sample from docs
Currently, `doc/source/_static/cloudkitty.conf.sample` is stored in
the git; however, building the doc overrides it. This is a problem
when building distro packages, as the clean target cannot be written
properly without hacks.

Change-Id: I28fb70e646b000032fb7181a3ffcc0d7097f9dc1
Story: #2010920
Task: #48780
2023-09-14 10:47:17 -03:00
OpenStack Release Bot 9b2e90dbde Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.

Sem-Ver: feature
Change-Id: I2d1191f45e036e9087618b7f3a2c2f758f28c85d
2023-09-14 01:42:50 +00:00
Rafael Weingärtner 60077a3cc4 Fix docs jobs in the CI that were broken due to Sphinx upgrade
Change-Id: Ic6278162fefdbfcb2ad6d7d07478b6a1e90c461e
2023-09-12 06:59:58 -03:00
Rafael Weingärtner 94630b97cd Use DevStack VENV path
DevStack is moving to adopt VENV to manage its python versions.
However, CloudKittty DevStack integration was not using the
DevStack VENV variable. This, in turn, causes issues with tempest
tests, as they are based on a DevStack deployment.

We need to merge this patch to fix the tempest tests.

Change-Id: I17de617557fb86c002814941325d71e3c08e0e72
2023-08-23 17:01:01 -03:00
Thomas Goirand b460fd937f Fix random unit test failures
As per https://bugs.debian.org/1029646, Cloudkitty often fails to build
as it fails its unit tests during the package build. This error happens
randomly. Sometimes it fails, sometimes it does not fail, but it's
clearly a false positive, because we don't really want the test to fail
in such case.

This patch makes it a lot less likely (10 times less) to happen by
increasing the tolerance.

Change-Id: If217a639f9af1e2693e6a132e46033df6bf96415
2023-07-06 08:46:45 +02:00
Zuul f2e70ca36f Merge "Allows multiple rating types for same metric for gnocchi" 2023-06-12 14:35:27 +00:00
Rafael Weingärtner 8c1ad7d0c6 Create concepts definition page for CloudKitty
Change-Id: I55f1f2e0f792943fed74240fa6eb474f14f6aec2
2023-05-29 17:23:24 +02:00
Zuul 5d024fc33f Merge "Remove `state` field from API" 2023-04-17 15:25:03 +00:00
Zuul 8520ddf824 Merge "Improve reprocessing task documentation" 2023-04-17 15:07:11 +00:00