As per the current release tested runtime, we test
till python 3.11 so updating the same in python
classifier in setup.cfg
Change-Id: If66420bae76e8db527c1aa191cb31d6ed8a7ff91
To facilitate the switch from Elasticsearch to OpenSearch, the ES
backend has been duplicated and renamed where appropriate to OpenSearch.
The OpenSearch implementation was modified in places for compatibility
with OpenSearch 2.x, for example:
- remove mapping name from bulk API URL
- replace put_mapping by post_mapping
This will allow for the future removal of the Elasticsearch backend.
Change-Id: I88b0a30f66af13dad1bd75cde412d2880b4ead30
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
Setuptools v54.1.0 introduces a warning that the use of dash-separated
options in 'setup.cfg' will not be supported in a future version [1].
Get ahead of the issue by replacing the dashes with underscores. Without
this, we see 'UserWarning' messages like the following on new enough
versions of setuptools:
UserWarning: Usage of dash-separated 'description-file' will not be
supported in future versions. Please use the underscore name
'description_file' instead
[1] https://github.com/pypa/setuptools/commit/a2e9ae4cb
Change-Id: I5e5e86149c0ea6be5553bf8911dfc729d0ec57bb
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also replace policy.json to policy.yaml ref from doc.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Change-Id: I608d3f55dfa9b6052f92c4fd13f2aae6d714e287
This introduces a v2 storage driver for ElasticSearch. It is considered
experimental for now and should not be used in production.
Support for this storage backend will be added to the devstack plugin.
Change-Id: I2f962a32ed541e62fcb847018f270613a69c8677
Story: 2006332
Task: 36076
Since data frames are now handled as objects, transformers are no longer
required. This simplifies the global codebase.
Story: 2005890
Task: 36075
Change-Id: I76d9117bd95d80e51ca95804c999f145e65c3a2d
A Prometheus scope fetcher has been added in order to dynamically discover
scopes from a Prometheus service using a user defined metric and a scope
attribute.
It can also filter out the response from Prometheus using metadata filters
to have a more fine-grained control over scope discovery.
It features HTTP basic auth capabilities and HTTPS configuration options
similar to Prometheus collector.
Change-Id: If3c2da8d7949e0aec08f3699547faf34af4ddee4
Story: 2005427
Task: 30458
This adds the ``cloudkitty-status upgrade check`` command to CloudKitty.
For now, this tool checks the storage version and raises a warning in case
v1 is used.
Depends-On: https://review.openstack.org/#/c/615928/
Change-Id: I39dc98fb716392a22765f169e2da0d389b33b941
Story: 2003657
Task: 26124
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.
This storage backend was only present for development purposes,
and not production ready. A second v2 backend will be implemented
in the future, with support for HA/clustering.
Change-Id: Iab9d152d2851ca385e607d338c0a09b74ba7e3b3
Story: 2004400
Task: 28568
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.
Change-Id: I60831f714c8a904170493a13bc82108d37d2c87d
Story: 2004400
Task: 28033
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.
Change-Id: Ifcc469aecd54ec22fbf76f623dde6b431c7e963b
Story: 2004400
Task: 28579
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.
Change-Id: Id2878bc9885132306333a7c675f3ac029a27a580
Story: 2004400
Task: 28580
This adds an InfluxDB backend to v2 storage. It is much more performant than the
gnocchi backend, and adds support for grafana.
In order to avoid making this patch too big, the documentation will be updated
in another patch.
Support for InfluxDB installation in the devstack plugin will also be added in
another patch.
Change-Id: Icaa23acb1a4791aac0dd8afb122d561065193eea
Story: 2001372
Task: 24536
This adds a gnocchi fetcher, which allows to dynamically retrieve available
scopes. Documentation will be extended once #592329 is merged.
Story: 2003652
Task: 26065
Change-Id: I9095008ea00dde99cbacb8f716d2d6b600fd4931
This adds a v2 storage interface as well as a gnocchi backend for it to
cloudkitty. This new interface will allow us to implement the v2 API, which will
be more metric-oriented.
This new storage uses the new dataframe format ('desc' is now split up into
'groupby' and 'metadata', following the collect logic); and supports grouping,
filters and pagination.
Modifications:
* The tenant/scope state management isn't handled by the storage backend
anymore. A StateManager object has been added.
* All internal interactions with the storage backend use the v2 interface.
An adapter class has been added to ensure retrocompatibility with the
v1 interface.
* A base class for functional tests testing the v2 storage interface has been
added. It contains generic tests, which should allow easier testing for new
v2 backends.
* Some developer documentation explaining how to implement a v2 storage
backend has been added. User documentation will be updated once the v2
storage backend is considered stable.
The v1 is still the default storage version. Note that this new version is
considered unstable and should only be used for upstream development.
Change-Id: I5e9b95c79292582fab3017289d35ee310e35ffea
Story: 2001372
Task: 6585
Task: 14361
Task: 24535
* Add Prometheus Collector that supports Prometheus HTTP instant queries.
* Due to the design of Cloudkitty, only instant queries are supported as
the time window used by Prometheus can exceed the one used by Cloudkitty,
using range queries, especially if coupled with function that
makes calculation back in time from a supplied timestamp
(e.g. increase(), delta()).
Change-Id: I4ba137b0b079f5ae6bfb645372778698eaa391fc
This patch provides a refactoring of the metric
configuration model (and file description) to
improve genericity, maintainability and usage for
non-openstack deployment.
The new metric yaml format is defined in the
attached story task and is validated on load with
voluptuous.
Now, a processor is dedicated to one collector and
one storage backend. Thus, collector and storage
configuration go back to the cloudkitty oslo conf.
Collectors have been refactored to have a code as similar as possible,
in order to ease comprehension for new contributors.
Story: 2001883
Task: 14354
Task: 14355
Task: 14431
Change-Id: I948dd9cd5c113bdaa4e49c532354938ffb45f0e7
* Add a SourceFetcher, which allows to retrieve new type of data sources.
It allows new collectors to be added.
(examples: Prometheus, Influxdb, Kubernetes...)
* Base Cloudkitty on metrics instead of services and resources
for collecting metrics. This new architecture allows Cloudkitty
to be more agnostic and so to rate containers metrics as the same way
as virtual machines metrics.
* Centralize metrology information in metrics.yml under metrics names.
Task: 6291
Story: 2001501
Change-Id: I00ca080cf05dfc03a3363720f85b79e003eda9be
As announced during the Queens cycle, all ceilometer-collector
related code has been removed.
Change-Id: I03f9a89ae4bc970fbc421dd6964be95cdacfe758
Task: 6293
Story: 2001503
As announced during the Queens cycle, the gnocchi and gnocchihybrid storage
backends have been removed.
Change-Id: I7654721cfaf7a48be8789ae4eb6939b4910ec9db
Task: 6294
Story: 2001503
This adds an hybrid storage backend to CloudKitty. It aims at splitting the
processor state management and the dataframe storage part. The default backend
for this storage is gnocchi, but other backends will be implemented in the
future.
This storage backend is a standard storage module for now, but it will become
the default storage once it is considered stable. Once this new storage type
has become the default, storage backend creation will be easier (no state
handling).
Change-Id: I61c0c17230350b12be3484ea4b5805960aa33099
Story: 2001372
This patch introduces the implementation for registering
default policy rules in code. Default rules are defined under
cloudkitty.common.policies. Each API's policies are defined in a
sub-folder under that path and __init__.py contains all the
default policies in code which are registered in the ``init``
enforcer function in cloudkitty/common/policy.py.
This commit does the following:
- Creates the ``policies`` module that contains all the default
policies in code.
- Adds the base policy rules into code (context_is_admin,
admin_or_owner and default rules).
- Add policies in code for current APIs
- Add a tox env to generate default policy sample file
- Delete policy.json from repo as policies in code will be used.
Change-Id: I257e8cefc2b699fc979c717531cd9ba77233d94b
Implements: blueprint policy-in-code
This adds a collector for Monasca. For now, only ceilometer metrics
published by ceilosca (https://github.com/openstack/monasca-ceilometer)
are collected, but this should be extended in the future.
Change-Id: If4553dd1ab1a45846699735979b54426e121a0b1
Task: 5717
Story: 2001211
Recommands to setup cloudkitty through an other WSGI services
like Apache 'mod_wsgi'. And the community has set a community wide goal
in Pike cycle: "Control Plane API endpoints deployment via WSGI"
https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html
Work Item: Add WSGI support
1. Provide WSGI application script file.
2. Removing the cloudkitty-api command line.
3. Adding cloudkitty-api wsgi_scripts, by 'cloudkitty-api -p 8889' to run.
Work Item: Make the devstack setup ck-api with wsgi
1. Switch devstack jobs to deploy control-plane API services in WSGI with Apache.
2. Default to deploy with Apache by global ENABLE_HTTPD_MOD_WSGI_SERVICES, in
local.conf expose CLOUDKITTY_USE_MOD_WSGI=False to run without Apache.
Work Item: Update the docs about installation
1. Installing the cloudkitty-api behind mod_wsgi.
2. Updating the installation about the cloudkitty-api.
Implements: blueprint wsgi-support
Change-Id: I207587c5360bb80c0e856cd0239e4073578951aa
The gating on python 3.4 is restricted to <= Mitaka. This is due
to the change from Ubuntu Trusty to Xenial, where only python3.5
is available. There is no need to continue to keep these settings.
Change-Id: I49007c51723bfbe7747bcc611ff669f3c7980a10
This storage driver adds full native support for gnocchi, improving the
performance compared with the hybrid solution and taking advantage of
gnocchi capabilities.
Change-Id: I65293ead2696967b028f8f9f11ecb84fb93380ff
Co-Authored-By: Stéphane Albert <stephane.albert@objectif-libre.com>
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the
convenience py35 venv.
Change-Id: Id43c6bf575aaa5e60e5161d1b6129382979468f3
This feature makes it possible to get your reports in CSV format.
It is now possible to set "pipeline" to csv in the [output] section
of cloudkitty.conf.
Co-Authored-By: Stéphane Albert <sheeprine@nullplace.com>
Change-Id: Ibba250462852906d43534cadf3f62f295c3136db
This storage removes the need to duplicate the data present in gnocchi.
Every dataframe is associated to the metric_id that generated the
calculations.
The downside is that you can only use gnocchi with this storage driver.
Moved around some utility function to remove the need to duplicate code
and have a cleaner boundary between the different parts of the code.
Change-Id: Iabfdcd4c15c906ed145ce383b65d1538f72671aa
CloudKitty now supports gnocchi data collection.
Support is still experimental as it's lacking some data.
Some fetched data might be inaccurate as it highly depend on your
archive policies.
Change-Id: I7e6668c766b7bd4641cccc2bd841a7aed1d2e2d5
This adds the CORS support middleware to CloudKitty, allowing a deployer
to optionally configure rules under which a javascript client may
break the single-origin policy and access the API directly.
For CloudKitty, we used the same gabbi fixtures and harnesses created for
ceilometer. A custom gabbi-paste.ini was created so that the API tests run
against the full wsgi application, and appropriate fixtures and tests were
created to correctly simulate the configuration state.
Hooks to ensure that cloudkitty's required HTTP headers are represented
both in the runtime defaults, and in the automatically generated config
file, have also been added.
OpenStack CrossProject Spec:
http://specs.openstack.org/openstack/openstack-specs/specs/cors-support.html
Oslo_Middleware Docs:
http://docs.openstack.org/developer/oslo.middleware/cors.html
OpenStack Cloud Admin Guide:
http://docs.openstack.org/admin-guide-cloud/cross_project_cors.html
DocImpact: Add link to CORS configuration in admin cloud guide.
Change-Id: I3ef96ca6c78c5e369fb09425871d0a57bf15ad8a