Commit Graph

70 Commits

Author SHA1 Message Date
Rafael Weingärtner b46dcd8983 Removal of Monasca fetcher and collector
Change-Id: I314c751f2b2dea693bea66d7d9b06679a06b0b6b
2024-02-14 08:59:28 -03:00
Ghanshyam Mann 9b2d97c6d4 Update python classifier in setup.cfg
As per the current release tested runtime, we test
till python 3.11 so updating the same in python
classifier in setup.cfg

Change-Id: If66420bae76e8db527c1aa191cb31d6ed8a7ff91
2024-01-28 22:40:42 +00:00
Matt Crees 964c6704a2 Add OpenSearch as a v2 storage backend
To facilitate the switch from Elasticsearch to OpenSearch, the ES
backend has been duplicated and renamed where appropriate to OpenSearch.

The OpenSearch implementation was modified in places for compatibility
with OpenSearch 2.x, for example:

- remove mapping name from bulk API URL
- replace put_mapping by post_mapping

This will allow for the future removal of the Elasticsearch backend.

Change-Id: I88b0a30f66af13dad1bd75cde412d2880b4ead30
Co-Authored-By: Pierre Riteau <pierre@stackhpc.com>
2023-11-27 15:19:14 +01:00
Ghanshyam Mann 73b256d849 Updating python testing classifier as per Yoga testing runtime
Yoga testing runtime[1] has been updated to add py39
testing as voting. Unit tests update are handled by the
job template change in openstack-zuul-job

- https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/820286

this commit updates the classifier in setup.cfg file.

[1] https://governance.openstack.org/tc/reference/runtimes/yoga.html

Change-Id: Ia3ad0dc42445548d17b83397288042ae33961da3
2021-12-14 02:47:24 +00:00
Pierre Riteau ebbbebc1d5 setup.cfg: Replace dashes with underscores
Setuptools v54.1.0 introduces a warning that the use of dash-separated
options in 'setup.cfg' will not be supported in a future version [1].
Get ahead of the issue by replacing the dashes with underscores. Without
this, we see 'UserWarning' messages like the following on new enough
versions of setuptools:

  UserWarning: Usage of dash-separated 'description-file' will not be
  supported in future versions. Please use the underscore name
  'description_file' instead

[1] https://github.com/pypa/setuptools/commit/a2e9ae4cb

Change-Id: I5e5e86149c0ea6be5553bf8911dfc729d0ec57bb
2021-06-07 20:30:35 +00:00
Ghanshyam Mann c8370c2ab2 [goal] Deprecate the JSON formatted policy file
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:

1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.

2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.

Also replace policy.json to policy.yaml ref from doc.

[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html

Change-Id: I608d3f55dfa9b6052f92c4fd13f2aae6d714e287
2020-12-22 18:49:31 -06:00
Quentin Anglade 8962a9ab63 Add a Monasca fetcher
This adds a Monasca fetcher for CloudKitty.

Change-Id: I0bf5af21987878d1e078021b0a9fbd0cc0a37b9d
Story: 2006675
Task: 36949
2020-09-09 15:09:09 -03:00
kangyufei b776aaf98e Add py38 package metadata
Change-Id: Ibf2290599ac9a0839b01e5df9a2b9a892ec0afe7
2020-09-07 12:27:15 +00:00
Andreas Jaeger 56609c46f9 [ussuri][goal] Cleanup drop python 2.7 support
OpenStack is dropping the py2.7 support in ussuri cycle.

Complete discussion & schedule can be found in
- http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.html
- https://etherpad.openstack.org/p/drop-python2-support

Ussuri Communtiy-wide goal:
https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html

this removes some cruft that can be removed now:
* cruft in setup.cfg and setup.py
* tox.ini: Add ignore_basepython_conflict and increase minversion
  for that
* Python 2/3 specific requirements, the repo is Python 3 only.

Change-Id: Iae68e58c82a180eb771bef00a3f44fd28abe5fae
2020-02-24 20:16:56 +01:00
Ghanshyam Mann afef05aca4 [ussuri][goal] Drop python 2.7 support and testing
OpenStack is dropping the py2.7 support in ussuri cycle.

Cloudkitty is ready with python 3 and ok to drop the
python 2.7 support.

Complete discussion & schedule can be found in
- http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.html
- https://etherpad.openstack.org/p/drop-python2-support

Ussuri Communtiy-wide goal:
https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html

Depends-On: https://review.opendev.org/#/c/693631/
Change-Id: I44ea5cfe479558969ccb3ca57f416511614626cc
2019-11-16 14:35:39 +00:00
Luka Peschke 15f6118ece Add an ElasticSearch v2 storage driver
This introduces a v2 storage driver for ElasticSearch. It is considered
experimental for now and should not be used in production.

Support for this storage backend will be added to the devstack plugin.

Change-Id: I2f962a32ed541e62fcb847018f270613a69c8677
Story: 2006332
Task: 36076
2019-09-13 15:29:40 +02:00
Luka Peschke 492ec063a7 Remove transformers from the codebase
Since data frames are now handled as objects, transformers are no longer
required. This simplifies the global codebase.

Story: 2005890
Task: 36075
Change-Id: I76d9117bd95d80e51ca95804c999f145e65c3a2d
2019-09-10 14:51:02 +00:00
pengyuesheng f34624cbf9 Add python 3.7 classifier to setup.cfg
Change-Id: I96e00b371bae337b21ef763b5bdc45eb3ce58f12
2019-06-12 11:05:44 +08:00
Justin Ferrieu 46a54ad05f Implement Prometheus fetcher
A Prometheus scope fetcher has been added in order to dynamically discover
scopes from a Prometheus service using a user defined metric and a scope
attribute.
It can also filter out the response from Prometheus using metadata filters
to have a more fine-grained control over scope discovery.
It features HTTP basic auth capabilities and HTTPS configuration options
similar to Prometheus collector.

Change-Id: If3c2da8d7949e0aec08f3699547faf34af4ddee4
Story: 2005427
Task: 30458
2019-04-17 12:18:51 +00:00
Ghanshyam Mann 19425bdfa1 Dropping the py35 testing
All the integration testing has been moved to
Bionic now[1] and py3.5 is not tested runtime for
Train or stable/stein[2].

As per below ML thread, we are good to drop the py35
testing now:
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005097.html

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004647.html
[2]
https://governance.openstack.org/tc/reference/runtimes/stein.html
https://governance.openstack.org/tc/reference/runtimes/train.html

Change-Id: I607d71e805a86098f8876ff35ad4443dc840740a
2019-04-14 19:18:28 +00:00
Zuul b2ee15e2c2 Merge "Support upgrade checks" 2019-03-07 19:36:03 +00:00
Luka Peschke b0487b6390 Support upgrade checks
This adds the ``cloudkitty-status upgrade check`` command to CloudKitty.
For now, this tool checks the storage version and raises a warning in case
v1 is used.

Depends-On: https://review.openstack.org/#/c/615928/
Change-Id: I39dc98fb716392a22765f169e2da0d389b33b941
Story: 2003657
Task: 26124
2019-02-04 08:42:21 +00:00
Luka Peschke 1ede03ba2c Delete v2 gnocchi storage
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.

This storage backend was only present for development purposes,
and not production ready. A second v2 backend will be implemented
in the future, with support for HA/clustering.

Change-Id: Iab9d152d2851ca385e607d338c0a09b74ba7e3b3
Story: 2004400
Task: 28568
2019-02-01 16:53:05 +01:00
Luka Peschke 2d7415a3d3 Remove the fake and meta collectors
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.

Change-Id: I60831f714c8a904170493a13bc82108d37d2c87d
Story: 2004400
Task: 28033
2019-01-31 12:39:43 +00:00
Zuul dafd27011a Merge "Remove the gnocchi transformer" 2019-01-31 12:04:21 +00:00
Luka Peschke 6d1b329a42 Remove the fake fetcher
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.

Change-Id: Ifcc469aecd54ec22fbf76f623dde6b431c7e963b
Story: 2004400
Task: 28579
2019-01-30 10:20:44 +00:00
Luka Peschke 6eb90404e4 Remove the gnocchi transformer
This is part of a global effort to clean up CloudKitty's
unmaintained codebase.

Change-Id: Id2878bc9885132306333a7c675f3ac029a27a580
Story: 2004400
Task: 28580
2019-01-30 10:20:32 +00:00
Luka Peschke c4758e78b4 Adding an InfluxDB storage backend
This adds an InfluxDB backend to v2 storage. It is much more performant than the
gnocchi backend, and adds support for grafana.

In order to avoid making this patch too big, the documentation will be updated
in another patch.

Support for InfluxDB installation in the devstack plugin will also be added in
another patch.

Change-Id: Icaa23acb1a4791aac0dd8afb122d561065193eea
Story: 2001372
Task: 24536
2018-12-11 14:32:55 +01:00
Justin Ferrieu 6350923970 Changed author email
From openstack-dev@lists.openstack.org to openstack-discuss@lists.openstack.org

Change-Id: Iccc4c8b416ecf2f32114de451a34ebc1e9494d1b
2018-12-06 09:45:17 +01:00
Luka Peschke 2b695d8fa5 Add a gnocchi fetcher
This adds a gnocchi fetcher, which allows to dynamically retrieve available
scopes. Documentation will be extended once #592329 is merged.

Story: 2003652
Task: 26065

Change-Id: I9095008ea00dde99cbacb8f716d2d6b600fd4931
2018-09-05 10:52:03 +02:00
Luka Peschke 3ab4774f66 Adding a v2 storage backend
This adds a v2 storage interface as well as a gnocchi backend for it to
cloudkitty. This new interface will allow us to implement the v2 API, which will
be more metric-oriented.

This new storage uses the new dataframe format ('desc' is now split up into
'groupby' and 'metadata', following the collect logic); and supports grouping,
filters and pagination.

Modifications:

  * The tenant/scope state management isn't handled by the storage backend
    anymore. A StateManager object has been added.

  * All internal interactions with the storage backend use the v2 interface.
    An adapter class has been added to ensure retrocompatibility with the
    v1 interface.

  * A base class for functional tests testing the v2 storage interface has been
    added. It contains generic tests, which should allow easier testing for new
    v2 backends.

  * Some developer documentation explaining how to implement a v2 storage
    backend has been added. User documentation will be updated once the v2
    storage backend is considered stable.

The v1 is still the default storage version. Note that this new version is
considered unstable and should only be used for upstream development.

Change-Id: I5e9b95c79292582fab3017289d35ee310e35ffea
Story: 2001372
Task: 6585
Task: 14361
Task: 24535
2018-08-22 12:11:25 +02:00
Martin CAMEY 49d18e168a Add Prometheus Collector
* Add Prometheus Collector that supports Prometheus HTTP instant queries.

* Due to the design of Cloudkitty, only instant queries are supported as
  the time window used by Prometheus can exceed the one used by Cloudkitty,
  using range queries, especially if coupled with function that
  makes calculation back in time from a supplied timestamp
  (e.g. increase(), delta()).

Change-Id: I4ba137b0b079f5ae6bfb645372778698eaa391fc
2018-08-21 19:51:56 +02:00
Luka Peschke 059a940392 Improve metrics configuration
This patch provides a refactoring of the metric
configuration model (and file description) to
improve genericity, maintainability and usage for
non-openstack deployment.

The new metric yaml format is defined in the
attached story task and is validated on load with
voluptuous.

Now, a processor is dedicated to one collector and
one storage backend. Thus, collector and storage
configuration go back to the cloudkitty oslo conf.

Collectors have been refactored to have a code as similar as possible,
in order to ease comprehension for new contributors.

Story: 2001883
Task: 14354
Task: 14355
Task: 14431

Change-Id: I948dd9cd5c113bdaa4e49c532354938ffb45f0e7
2018-06-28 11:19:51 +00:00
Martin CAMEY dff3e97b12 Allow Cloudkitty to collect non-OpenStack metrics
* Add a SourceFetcher, which allows to retrieve new type of data sources.
  It allows new collectors to be added.
  (examples: Prometheus, Influxdb, Kubernetes...)

* Base Cloudkitty on metrics instead of services and resources
  for collecting metrics. This new architecture allows Cloudkitty
  to be more agnostic and so to rate containers metrics as the same way
  as virtual machines metrics.

* Centralize metrology information in metrics.yml under metrics names.

Task: 6291
Story: 2001501

Change-Id: I00ca080cf05dfc03a3363720f85b79e003eda9be
2018-04-17 11:55:37 +02:00
Zuul 4020004269 Merge "Remove Ceilometer collector and transformer" 2018-03-14 10:23:05 +00:00
Luka Peschke dd64aaa3d5 Remove Ceilometer collector and transformer
As announced during the Queens cycle, all ceilometer-collector
related code has been removed.

Change-Id: I03f9a89ae4bc970fbc421dd6964be95cdacfe758
Task: 6293
Story: 2001503
2018-03-07 09:14:02 +00:00
Luka Peschke 0943330153 Remove gnocchi and gnocchihybrid storage
As announced during the Queens cycle, the gnocchi and gnocchihybrid storage
backends have been removed.

Change-Id: I7654721cfaf7a48be8789ae4eb6939b4910ec9db
Task: 6294
Story: 2001503
2018-03-06 11:46:06 +01:00
Luka Peschke d2f7e06362 Refactor the storage backend
This adds an hybrid storage backend to CloudKitty. It aims at splitting the
processor state management and the dataframe storage part. The default backend
for this storage is gnocchi, but other backends will be implemented in the
future.

This storage backend is a standard storage module for now, but it will become
the default storage once it is considered stable. Once this new storage type
has become the default, storage backend creation will be easier (no state
handling).

Change-Id: I61c0c17230350b12be3484ea4b5805960aa33099
Story: 2001372
2018-01-26 15:46:57 +00:00
Jeremy Liu 7eca672645 Policy in code
This patch introduces the implementation for registering
default policy rules in code. Default rules are defined under
cloudkitty.common.policies. Each API's policies are defined in a
sub-folder under that path and __init__.py contains all the
default policies in code which are registered in the ``init``
enforcer function in cloudkitty/common/policy.py.

This commit does the following:
 - Creates the ``policies`` module that contains all the default
   policies in code.
 - Adds the base policy rules into code (context_is_admin,
   admin_or_owner and default rules).
 - Add policies in code for current APIs
 - Add a tox env to generate default policy sample file
 - Delete policy.json from repo as policies in code will be used.

Change-Id: I257e8cefc2b699fc979c717531cd9ba77233d94b
Implements: blueprint policy-in-code
2017-12-08 16:45:13 +00:00
Luka Peschke 8876785d18 Add a collector for Monasca
This adds a collector for Monasca. For now, only ceilometer metrics
published by ceilosca (https://github.com/openstack/monasca-ceilometer)
are collected, but this should be extended in the future.

Change-Id: If4553dd1ab1a45846699735979b54426e121a0b1
Task: 5717
Story: 2001211
2017-10-27 14:53:01 +02:00
Jenkins 069ea4ea23 Merge "Update URLs in documents according to document migration" 2017-08-16 16:30:21 +00:00
liangcui cb39503e5b Update URLs in documents according to document migration
Change-Id: I9c27cc92f9706adf9bc021e5df5ed4e85ba925e9
2017-07-21 15:06:21 +08:00
zhangguoqing 2dc80310ba Add WSGI support for `cloudkitty-api'
Recommands to setup cloudkitty through an other WSGI services
like Apache 'mod_wsgi'. And the community has set a community wide goal
in Pike cycle: "Control Plane API endpoints deployment via WSGI"
https://governance.openstack.org/tc/goals/pike/deploy-api-in-wsgi.html

Work Item: Add WSGI support
1. Provide WSGI application script file.
2. Removing the cloudkitty-api command line.
3. Adding cloudkitty-api wsgi_scripts, by 'cloudkitty-api -p 8889' to run.

Work Item: Make the devstack setup ck-api with wsgi
1. Switch devstack jobs to deploy control-plane API services in WSGI with Apache.
2. Default to deploy with Apache by global ENABLE_HTTPD_MOD_WSGI_SERVICES, in
   local.conf expose CLOUDKITTY_USE_MOD_WSGI=False to run without Apache.

Work Item: Update the docs about installation
1. Installing the cloudkitty-api behind mod_wsgi.
2. Updating the installation about the cloudkitty-api.

Implements: blueprint wsgi-support
Change-Id: I207587c5360bb80c0e856cd0239e4073578951aa
2017-07-18 12:55:11 +02:00
gengchc2 0117a59252 Remove support for py34
The gating on python 3.4 is restricted to <= Mitaka. This is due
to the change from Ubuntu Trusty to Xenial, where only python3.5
is available. There is no need to continue to keep these settings.

Change-Id: I49007c51723bfbe7747bcc611ff669f3c7980a10
2017-02-08 16:27:08 +08:00
Jenkins c99d066467 Merge "Add Python 3.5 classifier and venv for cloudkitty" 2016-11-15 12:22:27 +00:00
avnish 0296672c00 modify the home-page info with the developer documentation
Change-Id: I8b068585b479df059013e3e95a4332a500866fee
2016-09-20 14:33:29 +05:30
Jenkins 7f608c264e Merge "Added native gnocchi storage driver" 2016-09-15 20:08:33 +00:00
scolinas 49e8417f13 Added native gnocchi storage driver
This storage driver adds full native support for gnocchi, improving the
performance compared with the hybrid solution and taking advantage of
gnocchi capabilities.

Change-Id: I65293ead2696967b028f8f9f11ecb84fb93380ff
Co-Authored-By: Stéphane Albert <stephane.albert@objectif-libre.com>
2016-09-15 21:48:08 +02:00
gengchc2 3acacc7ab4 Add Python 3.5 classifier and venv for cloudkitty
Now that there is a passing gate job, we can claim support for
Python 3.5 in the classifier. This patch also adds the
convenience py35 venv.

Change-Id: Id43c6bf575aaa5e60e5161d1b6129382979468f3
2016-09-05 09:42:04 +08:00
Luka Peschke 1196440c2f Add CSV support to cloudkitty-writer
This feature makes it possible to get your reports in CSV format.

It is now possible to set "pipeline" to csv in the [output] section
of cloudkitty.conf.

Co-Authored-By: Stéphane Albert <sheeprine@nullplace.com>

Change-Id: Ibba250462852906d43534cadf3f62f295c3136db
2016-09-01 15:01:44 +02:00
Jenkins 56cdfe9a9e Merge "Added support for an hybrid gnocchi storage" 2016-03-26 06:33:22 +00:00
Stéphane Albert 563b69fcd0 Added support for an hybrid gnocchi storage
This storage removes the need to duplicate the data present in gnocchi.
Every dataframe is associated to the metric_id that generated the
calculations.
The downside is that you can only use gnocchi with this storage driver.
Moved around some utility function to remove the need to duplicate code
and have a cleaner boundary between the different parts of the code.

Change-Id: Iabfdcd4c15c906ed145ce383b65d1538f72671aa
2016-03-26 03:10:57 +01:00
Stéphane Albert f7641aaa9f Added gnocchi collector
CloudKitty now supports gnocchi data collection.
Support is still experimental as it's lacking some data.
Some fetched data might be inaccurate as it highly depend on your
archive policies.

Change-Id: I7e6668c766b7bd4641cccc2bd841a7aed1d2e2d5
2016-03-25 18:10:12 +01:00
Michael Krotscheck 98a13bc212 Added CORS support to CloudKitty
This adds the CORS support middleware to CloudKitty, allowing a deployer
to optionally configure rules under which a javascript client may
break the single-origin policy and access the API directly.

For CloudKitty, we used the same gabbi fixtures and harnesses created for
ceilometer. A custom gabbi-paste.ini was created so that the API tests run
against the full wsgi application, and appropriate fixtures and tests were
created to correctly simulate the configuration state.

Hooks to ensure that cloudkitty's required HTTP headers are represented
both in the runtime defaults, and in the automatically generated config
file, have also been added.

OpenStack CrossProject Spec:
   http://specs.openstack.org/openstack/openstack-specs/specs/cors-support.html
Oslo_Middleware Docs:
   http://docs.openstack.org/developer/oslo.middleware/cors.html
OpenStack Cloud Admin Guide:
   http://docs.openstack.org/admin-guide-cloud/cross_project_cors.html
DocImpact: Add link to CORS configuration in admin cloud guide.

Change-Id: I3ef96ca6c78c5e369fb09425871d0a57bf15ad8a
2016-03-25 08:50:54 +01:00
Jenkins fff5cff0f4 Merge "Removed version information from setup.cfg" 2015-10-22 12:01:55 +00:00