Change all functions to use ``get_last_processed_timestamp`` instead of
``get_state``, update the tests and remove the ``get_state`` function.
Change-Id: Iea704fc594f4b5201a1fff7d38a6c0bafb9be6f1
Add file to the reno documentation build to show release notes for
stable/2024.1.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2024.1.
Sem-Ver: feature
Change-Id: Ic60212f53d02906092c8656b1fab8d9bb84d54a8
This patch allows CloudKitty to use InfluxDB v2 with Flux queries. This
type of query uses less CPU and RAM to be processed in the InfluxDB
backend.
Change-Id: I8ee3c92776aa69afbede353981a5fcd65dd7d099
Depends-On: https://review.opendev.org/c/openstack/requirements/+/895629
Story: 2010863
Task: 48539
There is the need to add human-readable description to the metric
definition. This can then be used to create custom reports in the
`summary` GET API. The value has to be stored in the backend as
we do with the alt_name and unit of the metric as well.
Depends-On: https://review.opendev.org/c/openstack/cloudkitty/+/861786
Change-Id: Icea8d00eaf3343e59f0f7b2234754f6abcb23258
To facilitate the switch from Elasticsearch to OpenSearch, the ES
backend has been duplicated and renamed where appropriate to OpenSearch.
The OpenSearch implementation was modified in places for compatibility
with OpenSearch 2.x, for example:
- remove mapping name from bulk API URL
- replace put_mapping by post_mapping
This will allow for the future removal of the Elasticsearch backend.
Change-Id: I88b0a30f66af13dad1bd75cde412d2880b4ead30
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
Introduce new default groupby options: (i) time: to group data by
hourly; (ii) time-d: to group data by day of the year; (iii) time-w:
to group data by week of the year; (iv) time-m: to group data by month;
and, (v) time-y: to group data by year. If you have old data in
CloudKitty and you wish to use these group by methods, you will need
to reprocess the desired timeframe.
Story: #2009839
Task: #44438
Depends-On: https://review.opendev.org/c/x/wsme/+/893677
Change-Id: Iad296f54f6701af84e168796aec9b1033a2a8a2d
Calling GET /v2/task/reprocesses with python-cloudkittyclient was
returning Internal Server Error, with the following API trace:
File "/var/lib/kolla/venv/lib/python3.6/site-packages/cloudkitty/api/v2/task/reprocess.py", line 259, in get
order, ACCEPTED_GET_REPROCESSING_REQUEST_ORDERS)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
This was because http_exceptions.BadRequest was given multiple arguments
(similar to LOG.* methods) instead of a single string.
Another issue is that python-cloudkittyclient sends the "DESC" order
while the API only supports "desc" and "asc". Convert to lower case for
compatibility.
Change-Id: Id1145adff82bc9a01e4eb0f306f0bfa535142459
Currently, when a reprocessing task is scheduled, CloudKitty executes
the cleaning of the data for the reprocessing period in one hour
fashion (the default period). Therefore, for each one of the
timeframes, a delete query is sent to InfluxDB (when using it as a
backend). However, InfluxDB is not a very optimized time series database
for deletion; thus, this workflow generates quite some overhead and
slowness when reprocessing. If we clean right away the whole time
frame for the reprocessing task, and then we just reprocess it, it will
execute a single delete query in InfluxDB, which has a similar cost as
a delete to remove the data for a single time frame.
This patch optimized the reprocessing workflow to execute batch cleaning
of data in the storage backend of CloudKitty.
Change-Id: I8282f44ad837c71df0cb6c73776eafc7014ebedf
The option 'use_all_resource_revisions' is useful when using Gnocchi
with the patch introduced in [1]. That patch can cause queries to
return more than one entry per granularity (timespan), according to
the revisions a resource has. This can be problematic when using the
'mutate' option of Cloudkitty. Therefore, this option
('use_all_resource_revisions') allows operators to discard all
datapoints returned from Gnocchi, but the last one in the granularity
that is queried by CloudKitty. The default behavior is maintained,
which means, CloudKitty always uses all the data points returned.
However, when the 'mutate' option is not used, we need to sum all the
quantities and use this value with the latest version of the attributes
received. Otherwise, we will miss the complete accounting for the time
frame where the revision happened.
[1] https://github.com/gnocchixyz/gnocchi/pull/1059
Change-Id: I45bdaa3783ff483d49ecca70571caf529f3ccbc3
Gnocchi fixed its `aggregates` API with PR
https://github.com/gnocchixyz/gnocchi/pull/1059. Before that patch,
the `aggregates` API would only return the latest metadata for the
resource of the metric being handled. Therefore, for CloudKitty
processing and reprocessing, we would always have the possibility of
using the wrong attribute version to rate the computing resources.
With this patch we propose to always use the correct metadata for the
processing and reprocessing of CloudKitty. This means, we always use
the metadata for the timestamp that we are collecting at Gnocchi.
The patch was released under version 4.5.0 of Gnocchi.
Change-Id: I31bc2cdf620fb5c0f561dc9de8c10d7882895cce
It was discovered that in some situations the same reprocessing task
might be processed simultaneously by different workers, which can
lead to unnecessary processing. This was happening due to the use
of "current_reprocess_time" in the lock name, which would lead to
different locking name for some situations; for instance, when worker
start processing a brand new reprocessing task, and after reprocessing
a few time frames, the "current_reprocess_time" is updated, then when
other workers achieve the same locking moment, they would have a
different lock name for the same scope ID, and reprocess a scope
that is currently in reprocessing.
Change-Id: I487d0eeb1cedc162d44f8c879a27f924b5c76206
Add file to the reno documentation build to show release notes for
stable/2023.2.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.
Sem-Ver: feature
Change-Id: I2d1191f45e036e9087618b7f3a2c2f758f28c85d
Following the footsteps of [1], depending on the environment where
CloudKitty is applied, when using Gnocchi fetcher, if the environment is
considerably big, CloudKitty tends to take too much time loading the
scope IDs from the Gnocchi API. To reduce that process time, we adopted
a different approach to discover the scope IDs registered in Gnocchi.
This patch proposes a change in that process, building on top of [1] to
load the chunk of Gnocchi resources and execute the retrieval of the
scope ID and selecting only the unique ones right away. Then, in the
worst case scenario, we would only have 1000 resources being loaded to
memory. Furthermore, we use the ``search`` API method from Gnocchi to
filter resources that are from scopes that are not in the set already
loaded. Therefore, we do not need to go over all of the resources in
Gnocchi. We will only list all of the resources that do not have the
scope IDs already loaded.
This reduced the processing time to load scope IDs from about 5 minutes
to 40 seconds in an environment with 500 scopes and ~50,000 resources.
[1] https://review.opendev.org/c/openstack/cloudkitty/+/864269
Change-Id: I9fa8934c5c857bd0069bb8038423e0126739a310
Queries filtering on date fields are slow as they have to parse each
row. There are sometimes millions of rows to parse while only a few
thousands are necessary.
The following patch narrows data to process by filtering more on
frame_model.begin as a firtst step using a `BETWEEN` statement instead
of >=
Change-Id: I8acbc8946d9e001419f7bf5064fcebe0a0ae907a
Depends-On: Ia6908d13c91a02c47863ae6ac4b595ac98f9fd91
Add file to the reno documentation build to show release notes for
stable/2023.1.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.
Sem-Ver: feature
Change-Id: I8a54b347affe9b0fc158dcf055f187c7d2e36eb1
Problem description
===================
It is not possible to create multiple rating types
for the same metric in Gnocchi, which forces operators
to create multiple metrics for the same resource type in
Gnocchi to create different rating types in Cloudkitty for
the same resource type in Gnocchi.
Proposal
========
We propose to extend the Gnocchi collector to allow operators
to create multiple rating types for the same metric in Gnocchi.
Using this approach we can create, for example, a rating type
for software licenses in a running instance and another rating
type for the instance flavor; it can be implemented using only
one metric in Gnocchi which has the instance installed softwares
and flavor metadata.
Change-Id: I69d4ba14cc72ba55e47baa6fd372f2085e1124da
Depending on the environment where CloudKitty is applied, when using
Gnocchi fetcher, if the environment is too big, CloudKitty tends to
consume too much RAM. This happens because before retrieving the scope
IDs and filtering out only the unique ones, it loads all of the Gnocchi
resources to memory.
This patch proposes a change in that process, to load the chunk of
Gnocchi resources and execute the retrieval of the scope ID and
selecting only the unique ones right away. Then, in the worst case
scenario, we would only have 1000 resources being loaded to memory.
Change-Id: Ibcc6af5aa5cf544be9032e33d3ac90eb1f6b14ef
The PyScript process in CloudKitty has been broken for a very long
time. This patch introduces changes required to make it work again.
Change-Id: I143ee6aa4352903921d2ab7b8d8468aedbdd6911
This is the first step towards moving API ref/docs to
https://docs.openstack.org/api
The `conf.py` file is a copy of the file from `doc/source` and all the
other files simply need moving to the new location.
Change-Id: I9ecf84b53274d9b86f05800fc9816de275f3e9c5
Problem description
===================
The reprocess API is accepting time windows that are
not compatible with the configured collection period
which causes some reprocessings to be endless or
generating different values based on the time window
the user inputs.
Proposal
========
We propose to add a validation in the reprocess API
to deny users to schedule a reprocess using a not
compatible time window and suggest the nearest valid
time window that the user can schedule a reprocess.
Change-Id: I24745a612bbd4714a7793df1deced671c1d1c26a
Add file to the reno documentation build to show release notes for
stable/zed.
Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/zed.
Sem-Ver: feature
Change-Id: I9ef3677caa0d4ae0e2eccda075730ab6bc125812
This mutator can map arbitrary values to new values. This is useful with
metrics reporting resource status as their value, but multiple statuses
are billable.
Change-Id: I8fcb9f2aa4ef23432089bfd6351a9c03ce3cf941
Dawud
OpenStack Release Bot