* Update cloudkitty from branch 'master'
to 8def21cb9cb439d7fd302b8e86663a59cc53faee
- Merge "Remove get_state function and its references"
- Remove get_state function and its references
Change all functions to use ``get_last_processed_timestamp`` instead of
``get_state``, update the tests and remove the ``get_state`` function.
Change-Id: Iea704fc594f4b5201a1fff7d38a6c0bafb9be6f1
* Update cloudkitty from branch 'master'
to 7385fb563e0cab9405945ca5589e8c7a45c68a05
- Merge "Update example pyscript at documentation"
- Update example pyscript at documentation
Updated service name and data object entries that do not match
current default metrics.yml configuration
Change-Id: Ib74661a764b887d24dcfb4e4f22ef376b6d1a4ad
* Update cloudkitty from branch 'master'
to b611e609547e38f91e25b7290d643e06daf9fd94
- Merge "Update master for stable/2024.1"
- Update master for stable/2024.1
Add file to the reno documentation build to show release notes for
stable/2024.1.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.1.
Sem-Ver: feature
Change-Id: Ic60212f53d02906092c8656b1fab8d9bb84d54a8
* Update cloudkitty from branch 'master'
to eaadb1427ee2a99a403f5ed19ba5f028462933f9
- Merge "reno: Update master for unmaintained/victoria"
- reno: Update master for unmaintained/victoria
Update the victoria release notes configuration to build from
unmaintained/victoria.
Change-Id: I37026520ed55a632cb006aafff891b26673eea5f
* Update cloudkitty from branch 'master'
to 9a6e167721002524450d6b9d28136622f226a83a
- Merge "reno: Update master for unmaintained/wallaby"
- reno: Update master for unmaintained/wallaby
Update the wallaby release notes configuration to build from
unmaintained/wallaby.
Change-Id: I87f39794a1d91796745d3d55f84b60ca96e39090
* Update cloudkitty from branch 'master'
to 38b4e41b1ba3e4b3d4b570fd4eec7505ddd922be
- Merge "reno: Update master for unmaintained/xena"
- reno: Update master for unmaintained/xena
Update the xena release notes configuration to build from
unmaintained/xena.
Change-Id: I3f6386e6705a5cf4e03044f64347ec6b35c1d0ea
* Update cloudkitty from branch 'master'
to 2a0904db00886602ef6376ebdd46057428741a34
- Set Elastic/OpenSearch jobs as voting in Zuul
Given these backends are now used in many production environments, they
can no longer be considered experimental.
Change-Id: I9e9f3023bf2a50807540e69b764600c0c5f995d5
* Update cloudkitty from branch 'master'
to f2c103535e554c84fbc6f14aefee45ba84f23ca8
- Replace usage of LegacyEngineFacade
Switch to using oslo_db.sqlalchemy.enginefacade instead, as this is
required for SQLAlchemy 2.x support.
Change-Id: Ifcad28239b6907b8ca396d348cbfa54185355f68
* Update cloudkitty from branch 'master'
to 3e35eb05edd5b475645a64047dc9337348bc7755
- Merge "Add requests to requirements"
- Add requests to requirements
We use requests in several places in cloudkitty code, so we need to
depend on it directly instead of installing it through dependencies.
Change-Id: Ibedb9ddba61b39c80b1e1b3910a90468bdfc76ae
* Update cloudkitty from branch 'master'
to 85ad6904fdc7bfde05732179c3e86e26d1af06d3
- Merge "Add support to InfluxDB v2 as storage backend"
- Add support to InfluxDB v2 as storage backend
This patch allows CloudKitty to use InfluxDB v2 with Flux queries. This
type of query uses less CPU and RAM to be processed in the InfluxDB
backend.
Change-Id: I8ee3c92776aa69afbede353981a5fcd65dd7d099
Depends-On: https://review.opendev.org/c/openstack/requirements/+/895629
Story: 2010863
Task: 48539
* Update cloudkitty from branch 'master'
to 328307512b8c49e692ae4c208f0f61c10d1821b6
- Merge "Removal of Monasca fetcher and collector"
- Removal of Monasca fetcher and collector
Change-Id: I314c751f2b2dea693bea66d7d9b06679a06b0b6b
* Update cloudkitty from branch 'master'
to 9ca4a42189a0ccaf423785a8c50506b6abbe7129
- Merge "Fix devstack runprocess for cloudkitty api"
- Fix devstack runprocess for cloudkitty api
After the commit
8778c64776
The module PBR in openstack started to validating the parameters when
creating an embedded WSGI, now if invalid arguments are given
to the cloudkitty-api it raises an error as we are facing in the
devstack when using `CLOUDKITTY_USE_MOD_WSGI=False`:
cloudkitty-api[86126]: usage: cloudkitty-api [-h] [--port PORT] [--host IP] -- [passed options]
cloudkitty-api[86126]: cloudkitty-api: error: unrecognized arguments: --config-file=/etc/cloudkitty/cloudkitty.conf
This PR also extracts the upgrade database workflow to a function
to be used in grenade tests
Change-Id: Ifc1a8563a9efcae2abaa6f8eb036405a93ff296d
* Update cloudkitty from branch 'master'
to 6414edca51e077a83fdf3bac5d1cf56c113c0e0a
- reno: Update master for unmaintained/yoga
Update the yoga release notes configuration to build from
unmaintained/yoga.
Change-Id: I889b90c6cc3e569b77ebdfc6ea1d3fe8a47288ae
* Update cloudkitty from branch 'master'
to 9b2d97c6d473d7ca2757e61cfb1549f3ad12e685
- Update python classifier in setup.cfg
As per the current release tested runtime, we test
till python 3.11 so updating the same in python
classifier in setup.cfg
Change-Id: If66420bae76e8db527c1aa191cb31d6ed8a7ff91
* Update cloudkitty from branch 'master'
to 3aa31f871965dc155ba0d8b3fad13b1358f8959f
- Add description option to a rating metric definition
There is the need to add human-readable description to the metric
definition. This can then be used to create custom reports in the
`summary` GET API. The value has to be stored in the backend as
we do with the alt_name and unit of the metric as well.
Depends-On: https://review.opendev.org/c/openstack/cloudkitty/+/861786
Change-Id: Icea8d00eaf3343e59f0f7b2234754f6abcb23258
* Update cloudkitty from branch 'master'
to 8eb8c4931aa400c904c03e2d4cdb02e1103bcf61
- Merge "Add OpenSearch as a v2 storage backend"
- Add OpenSearch as a v2 storage backend
To facilitate the switch from Elasticsearch to OpenSearch, the ES
backend has been duplicated and renamed where appropriate to OpenSearch.
The OpenSearch implementation was modified in places for compatibility
with OpenSearch 2.x, for example:
- remove mapping name from bulk API URL
- replace put_mapping by post_mapping
This will allow for the future removal of the Elasticsearch backend.
Change-Id: I88b0a30f66af13dad1bd75cde412d2880b4ead30
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
* Update cloudkitty from branch 'master'
to bf208c517ef4bebbb7e32935655427b90cb098e1
- Merge "Add groupby options by different timeframes"
- Add groupby options by different timeframes
Introduce new default groupby options: (i) time: to group data by
hourly; (ii) time-d: to group data by day of the year; (iii) time-w:
to group data by week of the year; (iv) time-m: to group data by month;
and, (v) time-y: to group data by year. If you have old data in
CloudKitty and you wish to use these group by methods, you will need
to reprocess the desired timeframe.
Story: #2009839
Task: #44438
Depends-On: https://review.opendev.org/c/x/wsme/+/893677
Change-Id: Iad296f54f6701af84e168796aec9b1033a2a8a2d
* Update cloudkitty from branch 'master'
to 5ebb5258200606212f946b5b949b154c2a0ae321
- Merge "Clean up release note"
- Clean up release note
Release notes are aimed for end users or operators, not for developers.
Change-Id: I0fb936b9c986fa159f6a19fffaad031ea6ff6df3
* Update cloudkitty from branch 'master'
to c842ea6ab244b382f2fa90ffa2a60bd5cce885fb
- Merge "Optimize CloudKitty reprocessing process"
- Optimize CloudKitty reprocessing process
Currently, when a reprocessing task is scheduled, CloudKitty executes
the cleaning of the data for the reprocessing period in one hour
fashion (the default period). Therefore, for each one of the
timeframes, a delete query is sent to InfluxDB (when using it as a
backend). However, InfluxDB is not a very optimized time series database
for deletion; thus, this workflow generates quite some overhead and
slowness when reprocessing. If we clean right away the whole time
frame for the reprocessing task, and then we just reprocess it, it will
execute a single delete query in InfluxDB, which has a similar cost as
a delete to remove the data for a single time frame.
This patch optimized the reprocessing workflow to execute batch cleaning
of data in the storage backend of CloudKitty.
Change-Id: I8282f44ad837c71df0cb6c73776eafc7014ebedf
* Update cloudkitty from branch 'master'
to 6075cdce95bc32043e9cf4a4fbfa4b063e60f048
- Merge "Patch for `use_all_resource_revisions` option"
- Patch for `use_all_resource_revisions` option
The option 'use_all_resource_revisions' is useful when using Gnocchi
with the patch introduced in [1]. That patch can cause queries to
return more than one entry per granularity (timespan), according to
the revisions a resource has. This can be problematic when using the
'mutate' option of Cloudkitty. Therefore, this option
('use_all_resource_revisions') allows operators to discard all
datapoints returned from Gnocchi, but the last one in the granularity
that is queried by CloudKitty. The default behavior is maintained,
which means, CloudKitty always uses all the data points returned.
However, when the 'mutate' option is not used, we need to sum all the
quantities and use this value with the latest version of the attributes
received. Otherwise, we will miss the complete accounting for the time
frame where the revision happened.
[1] https://github.com/gnocchixyz/gnocchi/pull/1059
Change-Id: I45bdaa3783ff483d49ecca70571caf529f3ccbc3
* Update cloudkitty from branch 'master'
to fe3b8f2bdcd44f83d1b2b451172500da9d8050f2
- Merge "Remove the manual patch for WSME"
- Remove the manual patch for WSME
WSME was released. Therefore, we can remove this work around.
Change-Id: I1673258da4fa9bcd96db24d635d9f06e3fc4a6a2
* Update cloudkitty from branch 'master'
to 12347e16c899936e4d9d12c506a88b0e84386b64
- Fix retrieval of reprocessing tasks
Calling GET /v2/task/reprocesses with python-cloudkittyclient was
returning Internal Server Error, with the following API trace:
File "/var/lib/kolla/venv/lib/python3.6/site-packages/cloudkitty/api/v2/task/reprocess.py", line 259, in get
order, ACCEPTED_GET_REPROCESSING_REQUEST_ORDERS)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
This was because http_exceptions.BadRequest was given multiple arguments
(similar to LOG.* methods) instead of a single string.
Another issue is that python-cloudkittyclient sends the "DESC" order
while the API only supports "desc" and "asc". Convert to lower case for
compatibility.
Change-Id: Id1145adff82bc9a01e4eb0f306f0bfa535142459
* Update cloudkitty from branch 'master'
to b96b9afc68af583c5dbbd935bb96f3e41806bfcf
- Merge "Use correct metadata for metrics gathered from gnocchi"
- Use correct metadata for metrics gathered from gnocchi
Gnocchi fixed its `aggregates` API with PR
https://github.com/gnocchixyz/gnocchi/pull/1059. Before that patch,
the `aggregates` API would only return the latest metadata for the
resource of the metric being handled. Therefore, for CloudKitty
processing and reprocessing, we would always have the possibility of
using the wrong attribute version to rate the computing resources.
With this patch we propose to always use the correct metadata for the
processing and reprocessing of CloudKitty. This means, we always use
the metadata for the timestamp that we are collecting at Gnocchi.
The patch was released under version 4.5.0 of Gnocchi.
Change-Id: I31bc2cdf620fb5c0f561dc9de8c10d7882895cce
* Update cloudkitty from branch 'master'
to c3fff2d464db4890b5acd228a84c7b85fa2bc3c8
- Merge "Fix a concurrency issue when locking reprocessing tasks"
- Fix a concurrency issue when locking reprocessing tasks
It was discovered that in some situations the same reprocessing task
might be processed simultaneously by different workers, which can
lead to unnecessary processing. This was happening due to the use
of "current_reprocess_time" in the lock name, which would lead to
different locking name for some situations; for instance, when worker
start processing a brand new reprocessing task, and after reprocessing
a few time frames, the "current_reprocess_time" is updated, then when
other workers achieve the same locking moment, they would have a
different lock name for the same scope ID, and reprocess a scope
that is currently in reprocessing.
Change-Id: I487d0eeb1cedc162d44f8c879a27f924b5c76206
* Update cloudkitty from branch 'master'
to 52b51b004ddb0e7881fff55b6f20ce18ab9c984f
- Merge "Remove cloudkitty.conf.sample from docs"
- Remove cloudkitty.conf.sample from docs
Currently, `doc/source/_static/cloudkitty.conf.sample` is stored in
the git; however, building the doc overrides it. This is a problem
when building distro packages, as the clean target cannot be written
properly without hacks.
Change-Id: I28fb70e646b000032fb7181a3ffcc0d7097f9dc1
Story: #2010920
Task: #48780
* Update cloudkitty from branch 'master'
to 31f825d3af814e12cf276c2528168624b11f7312
- Merge "Optimizing SQL queries that filter on a time range"
- Optimizing SQL queries that filter on a time range
Queries filtering on date fields are slow as they have to parse each
row. There are sometimes millions of rows to parse while only a few
thousands are necessary.
The following patch narrows data to process by filtering more on
frame_model.begin as a firtst step using a `BETWEEN` statement instead
of >=
Change-Id: I8acbc8946d9e001419f7bf5064fcebe0a0ae907a
Depends-On: Ia6908d13c91a02c47863ae6ac4b595ac98f9fd91
* Update cloudkitty from branch 'master'
to 9b2e90dbde84b3578e47845c0e0b8552d560ecff
- Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.
Sem-Ver: feature
Change-Id: I2d1191f45e036e9087618b7f3a2c2f758f28c85d
* Update cloudkitty from branch 'master'
to 60077a3cc47eaa49e33930bfe4253bff784c6a63
- Fix docs jobs in the CI that were broken due to Sphinx upgrade
Change-Id: Ic6278162fefdbfcb2ad6d7d07478b6a1e90c461e
* Update cloudkitty from branch 'master'
to 94630b97cd1fb4bdd9a638070ffbbe3625de8aa2
- Use DevStack VENV path
DevStack is moving to adopt VENV to manage its python versions.
However, CloudKittty DevStack integration was not using the
DevStack VENV variable. This, in turn, causes issues with tempest
tests, as they are based on a DevStack deployment.
We need to merge this patch to fix the tempest tests.
Change-Id: I17de617557fb86c002814941325d71e3c08e0e72
* Update cloudkitty from branch 'master'
to b460fd937fd2c8cc83979efa1abb7fa82d40a3f7
- Fix random unit test failures
As per https://bugs.debian.org/1029646, Cloudkitty often fails to build
as it fails its unit tests during the package build. This error happens
randomly. Sometimes it fails, sometimes it does not fail, but it's
clearly a false positive, because we don't really want the test to fail
in such case.
This patch makes it a lot less likely (10 times less) to happen by
increasing the tolerance.
Change-Id: If217a639f9af1e2693e6a132e46033df6bf96415
* Update cloudkitty from branch 'master'
to f2e70ca36f6159861ef792ab3a7c07ef4f559303
- Merge "Allows multiple rating types for same metric for gnocchi"
- Allows multiple rating types for same metric for gnocchi
Problem description
===================
It is not possible to create multiple rating types
for the same metric in Gnocchi, which forces operators
to create multiple metrics for the same resource type in
Gnocchi to create different rating types in Cloudkitty for
the same resource type in Gnocchi.
Proposal
========
We propose to extend the Gnocchi collector to allow operators
to create multiple rating types for the same metric in Gnocchi.
Using this approach we can create, for example, a rating type
for software licenses in a running instance and another rating
type for the instance flavor; it can be implemented using only
one metric in Gnocchi which has the instance installed softwares
and flavor metadata.
Change-Id: I69d4ba14cc72ba55e47baa6fd372f2085e1124da
* Update cloudkitty from branch 'master'
to 5d024fc33fc4d8ce6500d16668b9857aee49bd85
- Merge "Remove `state` field from API"
- Remove `state` field from API
Following the patch [1], this patchset executes the removal of the state
field from the API.
[1] https://review.opendev.org/c/openstack/cloudkitty/+/774634
Change-Id: I0dfef93c603524b732e6b2694d9c8877826b5ddc
* Update cloudkitty from branch 'master'
to ec4ae720f4895c545a683ec2fcf93e477229a433
- Merge "Create indexes to allow SQL rewrites and optimizations"
- Create indexes to allow SQL rewrites and optimizations
Change-Id: Ia6908d13c91a02c47863ae6ac4b595ac98f9fd91
* Update cloudkitty from branch 'master'
to d7c70847b66bb6b7b3ff61ed2b690eaeb8c3a3b4
- Merge "Optimize Gnocchi fetcher processing time"
- Optimize Gnocchi fetcher processing time
Following the footsteps of [1], depending on the environment where
CloudKitty is applied, when using Gnocchi fetcher, if the environment is
considerably big, CloudKitty tends to take too much time loading the
scope IDs from the Gnocchi API. To reduce that process time, we adopted
a different approach to discover the scope IDs registered in Gnocchi.
This patch proposes a change in that process, building on top of [1] to
load the chunk of Gnocchi resources and execute the retrieval of the
scope ID and selecting only the unique ones right away. Then, in the
worst case scenario, we would only have 1000 resources being loaded to
memory. Furthermore, we use the ``search`` API method from Gnocchi to
filter resources that are from scopes that are not in the set already
loaded. Therefore, we do not need to go over all of the resources in
Gnocchi. We will only list all of the resources that do not have the
scope IDs already loaded.
This reduced the processing time to load scope IDs from about 5 minutes
to 40 seconds in an environment with 500 scopes and ~50,000 resources.
[1] https://review.opendev.org/c/openstack/cloudkitty/+/864269
Change-Id: I9fa8934c5c857bd0069bb8038423e0126739a310
* Update cloudkitty from branch 'master'
to 2153ece172c8fc10ffa8dbb41e9777390f29aadd
- Merge "Add detail about reprocessing all scope IDs in reprocessing API"
- Add detail about reprocessing all scope IDs in reprocessing API
Change-Id: Ic2f87d41f3c88cfb87e9d714cd2fb805897f57e9
* Update cloudkitty from branch 'master'
to f8de1a2b21083ba5214788b57eda7bafdc153f92
- Merge "Update master for stable/2023.1"
- Update master for stable/2023.1
Add file to the reno documentation build to show release notes for
stable/2023.1.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.
Sem-Ver: feature
Change-Id: I8a54b347affe9b0fc158dcf055f187c7d2e36eb1
* Update cloudkitty from branch 'master'
to 73d5543f15ec188fafa1562f4142edd41afbad5c
- Add missing "." for api-ref
Add missing "." for list of errors, all others have a "." except 404.
This change is also meant to help debug why api-ref is not published at
all. See
https://review.opendev.org/c/openstack/openstack-manuals/+/867357 for
details.
Change-Id: I69f8eab3c71a469f81561beac32b8193339237c0
* Update cloudkitty from branch 'master'
to 974b8b96074354fa9926c5bcbadf71276fc7370e
- Merge "Optimize Gnocchi fetcher"
- Optimize Gnocchi fetcher
Depending on the environment where CloudKitty is applied, when using
Gnocchi fetcher, if the environment is too big, CloudKitty tends to
consume too much RAM. This happens because before retrieving the scope
IDs and filtering out only the unique ones, it loads all of the Gnocchi
resources to memory.
This patch proposes a change in that process, to load the chunk of
Gnocchi resources and execute the retrieval of the scope ID and
selecting only the unique ones right away. Then, in the worst case
scenario, we would only have 1000 resources being loaded to memory.
Change-Id: Ibcc6af5aa5cf544be9032e33d3ac90eb1f6b14ef
* Update cloudkitty from branch 'master'
to aec309207e13aa83a33c7a5e6568a7c491d8bb8d
- Merge "Fix PyScripts processing"
- Fix PyScripts processing
The PyScript process in CloudKitty has been broken for a very long
time. This patch introduces changes required to make it work again.
Change-Id: I143ee6aa4352903921d2ab7b8d8468aedbdd6911
* Update cloudkitty from branch 'master'
to 97567939883fbe4b025630319fc9ab3bcaa5ae7b
- Use new get_rpc_client API from oslo.messaging
Use the new API that is consistent with
the existing API instead of instantiating the client
class directly.
This was introduced in release 14.1.0 here [1] and
added into oslo.messaging here [2]
[1] https://review.opendev.org/c/openstack/requirements/+/869340
[2] https://review.opendev.org/c/openstack/oslo.messaging/+/862419
Change-Id: Ia95d2c85cded3be8944c0cc5fafc30d37df9ce82
* Update cloudkitty from branch 'master'
to 1d90a83fc464ee002bb3488d242c78368a48fc93
- Merge "Add missing api-ref envlist to tox"
- Add missing api-ref envlist to tox
This is a follow-up to:
https://review.opendev.org/c/openstack/cloudkitty/+/867122
We need to add the `[testenv:api-ref]` section to `tox.ini` to be able
to run `tox -e api-ref`, which is what Zuul uses to build the API refs.
Sphinx had to be capped to the latest version 4 or otherwise `tox` fails
with the "add_content() takes 2 positional arguments but 3 were given"
error message.
Change-Id: I65b008152e2bc64f29229996c87ad4587fe85043
* Update cloudkitty from branch 'master'
to 17a1816da05aa7834670e2725369a8b04dd909c1
- Merge "Make tox.ini tox 4.0.0 compatible"
- Make tox.ini tox 4.0.0 compatible
* removed skipsdist=True to make sure cloudkitty is available in the
virtual env
* added find to allowed external commands in tox
* replaced full path to find for readability
Change-Id: I4211e9be3e170f8724246883768247727b1d5d0b
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
* Update cloudkitty from branch 'master'
to 3a90e20130917d3af0171e35cc9f7154c5b1706e
- Merge "Move API docs to `api-ref/source`"
- Move API docs to `api-ref/source`
This is the first step towards moving API ref/docs to
https://docs.openstack.org/api
The `conf.py` file is a copy of the file from `doc/source` and all the
other files simply need moving to the new location.
Change-Id: I9ecf84b53274d9b86f05800fc9816de275f3e9c5
Zuul
OpenStack Release Bot