Change all functions to use ``get_last_processed_timestamp`` instead of
``get_state``, update the tests and remove the ``get_state`` function.
Change-Id: Iea704fc594f4b5201a1fff7d38a6c0bafb9be6f1
Given these backends are now used in many production environments, they
can no longer be considered experimental.
Change-Id: I9e9f3023bf2a50807540e69b764600c0c5f995d5
Switch to using oslo_db.sqlalchemy.enginefacade instead, as this is
required for SQLAlchemy 2.x support.
Change-Id: Ifcad28239b6907b8ca396d348cbfa54185355f68
This patch allows CloudKitty to use InfluxDB v2 with Flux queries. This
type of query uses less CPU and RAM to be processed in the InfluxDB
backend.
Change-Id: I8ee3c92776aa69afbede353981a5fcd65dd7d099
Depends-On: https://review.opendev.org/c/openstack/requirements/+/895629
Story: 2010863
Task: 48539
There is the need to add human-readable description to the metric
definition. This can then be used to create custom reports in the
`summary` GET API. The value has to be stored in the backend as
we do with the alt_name and unit of the metric as well.
Depends-On: https://review.opendev.org/c/openstack/cloudkitty/+/861786
Change-Id: Icea8d00eaf3343e59f0f7b2234754f6abcb23258
To facilitate the switch from Elasticsearch to OpenSearch, the ES
backend has been duplicated and renamed where appropriate to OpenSearch.
The OpenSearch implementation was modified in places for compatibility
with OpenSearch 2.x, for example:
- remove mapping name from bulk API URL
- replace put_mapping by post_mapping
This will allow for the future removal of the Elasticsearch backend.
Change-Id: I88b0a30f66af13dad1bd75cde412d2880b4ead30
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
Introduce new default groupby options: (i) time: to group data by
hourly; (ii) time-d: to group data by day of the year; (iii) time-w:
to group data by week of the year; (iv) time-m: to group data by month;
and, (v) time-y: to group data by year. If you have old data in
CloudKitty and you wish to use these group by methods, you will need
to reprocess the desired timeframe.
Story: #2009839
Task: #44438
Depends-On: https://review.opendev.org/c/x/wsme/+/893677
Change-Id: Iad296f54f6701af84e168796aec9b1033a2a8a2d
Calling GET /v2/task/reprocesses with python-cloudkittyclient was
returning Internal Server Error, with the following API trace:
File "/var/lib/kolla/venv/lib/python3.6/site-packages/cloudkitty/api/v2/task/reprocess.py", line 259, in get
order, ACCEPTED_GET_REPROCESSING_REQUEST_ORDERS)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
This was because http_exceptions.BadRequest was given multiple arguments
(similar to LOG.* methods) instead of a single string.
Another issue is that python-cloudkittyclient sends the "DESC" order
while the API only supports "desc" and "asc". Convert to lower case for
compatibility.
Change-Id: Id1145adff82bc9a01e4eb0f306f0bfa535142459
Currently, when a reprocessing task is scheduled, CloudKitty executes
the cleaning of the data for the reprocessing period in one hour
fashion (the default period). Therefore, for each one of the
timeframes, a delete query is sent to InfluxDB (when using it as a
backend). However, InfluxDB is not a very optimized time series database
for deletion; thus, this workflow generates quite some overhead and
slowness when reprocessing. If we clean right away the whole time
frame for the reprocessing task, and then we just reprocess it, it will
execute a single delete query in InfluxDB, which has a similar cost as
a delete to remove the data for a single time frame.
This patch optimized the reprocessing workflow to execute batch cleaning
of data in the storage backend of CloudKitty.
Change-Id: I8282f44ad837c71df0cb6c73776eafc7014ebedf
The option 'use_all_resource_revisions' is useful when using Gnocchi
with the patch introduced in [1]. That patch can cause queries to
return more than one entry per granularity (timespan), according to
the revisions a resource has. This can be problematic when using the
'mutate' option of Cloudkitty. Therefore, this option
('use_all_resource_revisions') allows operators to discard all
datapoints returned from Gnocchi, but the last one in the granularity
that is queried by CloudKitty. The default behavior is maintained,
which means, CloudKitty always uses all the data points returned.
However, when the 'mutate' option is not used, we need to sum all the
quantities and use this value with the latest version of the attributes
received. Otherwise, we will miss the complete accounting for the time
frame where the revision happened.
[1] https://github.com/gnocchixyz/gnocchi/pull/1059
Change-Id: I45bdaa3783ff483d49ecca70571caf529f3ccbc3
Gnocchi fixed its `aggregates` API with PR
https://github.com/gnocchixyz/gnocchi/pull/1059. Before that patch,
the `aggregates` API would only return the latest metadata for the
resource of the metric being handled. Therefore, for CloudKitty
processing and reprocessing, we would always have the possibility of
using the wrong attribute version to rate the computing resources.
With this patch we propose to always use the correct metadata for the
processing and reprocessing of CloudKitty. This means, we always use
the metadata for the timestamp that we are collecting at Gnocchi.
The patch was released under version 4.5.0 of Gnocchi.
Change-Id: I31bc2cdf620fb5c0f561dc9de8c10d7882895cce
It was discovered that in some situations the same reprocessing task
might be processed simultaneously by different workers, which can
lead to unnecessary processing. This was happening due to the use
of "current_reprocess_time" in the lock name, which would lead to
different locking name for some situations; for instance, when worker
start processing a brand new reprocessing task, and after reprocessing
a few time frames, the "current_reprocess_time" is updated, then when
other workers achieve the same locking moment, they would have a
different lock name for the same scope ID, and reprocess a scope
that is currently in reprocessing.
Change-Id: I487d0eeb1cedc162d44f8c879a27f924b5c76206
As per https://bugs.debian.org/1029646, Cloudkitty often fails to build
as it fails its unit tests during the package build. This error happens
randomly. Sometimes it fails, sometimes it does not fail, but it's
clearly a false positive, because we don't really want the test to fail
in such case.
This patch makes it a lot less likely (10 times less) to happen by
increasing the tolerance.
Change-Id: If217a639f9af1e2693e6a132e46033df6bf96415
Following the footsteps of [1], depending on the environment where
CloudKitty is applied, when using Gnocchi fetcher, if the environment is
considerably big, CloudKitty tends to take too much time loading the
scope IDs from the Gnocchi API. To reduce that process time, we adopted
a different approach to discover the scope IDs registered in Gnocchi.
This patch proposes a change in that process, building on top of [1] to
load the chunk of Gnocchi resources and execute the retrieval of the
scope ID and selecting only the unique ones right away. Then, in the
worst case scenario, we would only have 1000 resources being loaded to
memory. Furthermore, we use the ``search`` API method from Gnocchi to
filter resources that are from scopes that are not in the set already
loaded. Therefore, we do not need to go over all of the resources in
Gnocchi. We will only list all of the resources that do not have the
scope IDs already loaded.
This reduced the processing time to load scope IDs from about 5 minutes
to 40 seconds in an environment with 500 scopes and ~50,000 resources.
[1] https://review.opendev.org/c/openstack/cloudkitty/+/864269
Change-Id: I9fa8934c5c857bd0069bb8038423e0126739a310
Queries filtering on date fields are slow as they have to parse each
row. There are sometimes millions of rows to parse while only a few
thousands are necessary.
The following patch narrows data to process by filtering more on
frame_model.begin as a firtst step using a `BETWEEN` statement instead
of >=
Change-Id: I8acbc8946d9e001419f7bf5064fcebe0a0ae907a
Depends-On: Ia6908d13c91a02c47863ae6ac4b595ac98f9fd91
Problem description
===================
It is not possible to create multiple rating types
for the same metric in Gnocchi, which forces operators
to create multiple metrics for the same resource type in
Gnocchi to create different rating types in Cloudkitty for
the same resource type in Gnocchi.
Proposal
========
We propose to extend the Gnocchi collector to allow operators
to create multiple rating types for the same metric in Gnocchi.
Using this approach we can create, for example, a rating type
for software licenses in a running instance and another rating
type for the instance flavor; it can be implemented using only
one metric in Gnocchi which has the instance installed softwares
and flavor metadata.
Change-Id: I69d4ba14cc72ba55e47baa6fd372f2085e1124da
Depending on the environment where CloudKitty is applied, when using
Gnocchi fetcher, if the environment is too big, CloudKitty tends to
consume too much RAM. This happens because before retrieving the scope
IDs and filtering out only the unique ones, it loads all of the Gnocchi
resources to memory.
This patch proposes a change in that process, to load the chunk of
Gnocchi resources and execute the retrieval of the scope ID and
selecting only the unique ones right away. Then, in the worst case
scenario, we would only have 1000 resources being loaded to memory.
Change-Id: Ibcc6af5aa5cf544be9032e33d3ac90eb1f6b14ef
The PyScript process in CloudKitty has been broken for a very long
time. This patch introduces changes required to make it work again.
Change-Id: I143ee6aa4352903921d2ab7b8d8468aedbdd6911
Problem description
===================
The reprocess API is accepting time windows that are
not compatible with the configured collection period
which causes some reprocessings to be endless or
generating different values based on the time window
the user inputs.
Proposal
========
We propose to add a validation in the reprocess API
to deny users to schedule a reprocess using a not
compatible time window and suggest the nearest valid
time window that the user can schedule a reprocess.
Change-Id: I24745a612bbd4714a7793df1deced671c1d1c26a
oslo.db 12.1.0 has changed the default value for the 'autocommit'
parameter of 'LegacyEngineFacade' from 'True' to 'False'. This is a
necessary step to ensure compatibility with SQLAlchemy 2.0. However, we
are currently relying on the autocommit behavior and need changes to
explicitly manage sessions. Until that happens, we need to override the
default.
Co-Authored-By: Stephen Finucane <stephenfin@redhat.com>
Change-Id: Ia0e9696dcaafd90f9c6daeb68c72fa2b184823fb
This mutator can map arbitrary values to new values. This is useful with
metrics reporting resource status as their value, but multiple statuses
are billable.
Change-Id: I8fcb9f2aa4ef23432089bfd6351a9c03ce3cf941
It was deprecated in favor of assertRaisesRegex added in Python 3.2.
We can replace it now because Python 2 is no longer supported.
Change-Id: I4cec6e44d48bcee9808bfd647f3a45cd8b1e2f11
Currently, CloudKitty only allows creating rating rules as
"99.999999999999999999999999". Therefore, for prices equal to or higher
than 100, we would not be able to use them. This patch will enable
operators to use any value between 0 and 999999999999 (in the integer
part of the number), which will provide more flexibility.
Change-Id: I2ff4a09ce3b0fdf0b08a7e565b58794b25ac5ade
Story: 2009947
Task: 44865
This commit adds an API enabling the POST operation to create scopes in
an ad hoc fashion. This is useful for operators to register scopes
before they are created as resources in the collected backend so their
processing can be discarded right away, for example for trial
projects/accounts.
Otherwise, we need to wait for them to create resources, then for
example Ceilometer has to monitor these resources, persist measures in
Gnocchi, then CloudKitty has to discover the scopes and finally we can
disable their processing.
Change-Id: I3e947d36c9d5d5da07115d35dde578ae300cbe5c
The voluptuous.Coerce validator expects a type, not a function. Use
tzutils.dt_from_iso directly instead, since voluptuous.Schema accepts
callables.
Change-Id: Ibe01ac289f00f24fa35ad2416fa2a93d8d6f57a6
Story: 2010058
Task: 45502
This introduces GET methods for rating modules in the v2 API. Work items:
* Implement the "/v1/rating/modules" endpoints in the v2 API, including
unit tests and documentation
Story: 2006572
Task: 36677
Co-Authored-By: Rafael Weingärtner <rafael@apache.org>
Change-Id: I0a2f24051d268a396955c8df7e3e5615546f6293
Dawud