Commit Graph

66 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov 10164be39e Add quorum queues support for service
This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.

In order to ensure upgrade path and ability to switch back to HA queues
we change vhost names with removing leading `/`, as enabling quorum
requires to remove exchange which is tricky thing to do with running
services.

Change-Id: I722aefb5290a68311faf1d44c279151e8492466a
2023-10-20 11:53:46 +00:00
Dmitriy Rabotyagov c7b2ad25c9 Fix linters and metadata
With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.

With that we also update metdata to reflect current state.

Depends-On: https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/888223
Change-Id: Ib51c7ec8d83ffe82c98d1c8d3200ce4510775895
2023-07-14 17:11:05 +00:00
Damian Dabrowski ba2e621a93 Add TLS support to cloudkitty backends
By overriding the variable `cloudkitty_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the cloudkitty backend api.

The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: I0566028b4737306954b544d040b52cee1fddaa1f
2023-04-29 18:36:21 +02:00
Dmitriy Rabotyagov b6d15a95cb Support service tokens
Implement support for service_tokens. For that we convert
role_name to be a list along with renaming corresponding variable.

Additionally service_type is defined now for keystone_authtoken which
enables to validate tokens with restricted access rules

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690
Change-Id: Icb1de8c7e0a5196a4df457a5d4a3ca524d4622d0
2022-06-15 19:17:22 +00:00
Damian Dabrowski 40d38d0000 Database connection pooling improvements
- Implemented new variable ``connection_recycle_time`` responsible for SQLAlchemy's connection recycling
- Set new default values for db pooling variables which are inherited from the global ones.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/819424
Change-Id: I4ae28b356a404c1e76e5cff986d4fe0172b564f2
2021-12-03 11:51:23 +01:00
Dmitriy Rabotyagov 319dadd643 Refactor galera_use_ssl behaviour
With PKI role in place in most cases you don't need to explicitly
provide path to the CA file because PKI role ensures that CA is trusted
by the system overall. In the meanwhile in PyMySQL [1] you must either
provide CA file or cert/key or enable verify.

Since current behaviour is to provide path to the custom CA we expect
certificate being trusted overall. Thus we enable cert verification when
galera_use_ssl is True.

[1] 78f0cf99e5/pymysql/connections.py (L267)

Change-Id: I756a6cdaa4b619671da73c65761c796a0b2ade81
2021-09-21 14:23:55 +03:00
Jonathan Herlin b67bd60df3 Cloudkitty role cleanup and config updates
The Cloudkitty role has been lacking some attention for a while
causing some of the configuration and examples outdated.

Now that Cloudkitty playbook and dashboard can be installed using
OSA without running any additional playbooks from this repository
i have removed those from here.

Change-Id: I0fe96c318273f0016d93cf043bda74feb11c63f8
2021-06-09 10:03:21 +02:00
Jonathan Rosser 68e8d15cd5 Add variables for rabbitmq ssl configuration
Change-Id: If8362185261aefa741f9c21c57b3dfe258d8543d
2021-05-17 09:24:44 +00:00
Jonathan Rosser 2b54345586 Use ansible_facts[] instead of fact variables
See https://github.com/ansible/ansible/issues/73654

Change-Id: Ia088d1b4074e3bbcd3054f5be00b39429437750d
2021-03-16 08:01:56 +00:00
Jonathan Rosser f0f23521bc Switch default virtualenv to python3
Change-Id: I0290f78f2eaf763238abdffacea6fd34c75afea8
2021-03-10 08:48:36 +00:00
Jonathan Rosser e0b8bd00a1 Move cloudkitty pip packages from constraints to requirements
This is necessary for the new pip resolver

Change-Id: I8d70caf213e3f4533fc7490451e95a85ff191ad8
2021-01-25 09:48:54 +00:00
Dmitriy Rabotyagov e84e327c2d Reduce number of processes on small systems
Even the most modest 4C/8T system would run with the maximum 16 processes
due to the calculation being VCPU*2.

We devide amount of CPUs to number of threads for hyperthreaded CPUs

Change-Id: Ifa09cdddcb4a3875f064ff773bbe2281f964fb1d
2020-10-30 18:05:33 +02:00
Dmitriy Rabotyagov 18ca2f1618 Use the utility host for db setup tasks
Move it to the service setup host (defaults to utility[0]) instead
of the galera[0] host, and use galera_address (defaults to internal VIP)
as the endpoint instead of a local connection on the db host.

Change-Id: Ie8e6b95f531fef9c9fe04d6d8f692d599aa5ff30
2020-08-27 18:09:04 +03:00
Dmitriy Rabotyagov 5eecaef17b Cleanup after repo_build and pip_install retirement
Change-Id: I6e39a492e1d8f73f530bc0928881fb156ff05aaf
2020-05-12 21:26:29 +03:00
Guilherme Steinmüller 9988a75734 Refactor memcached_servers
This patch aims to add a prefix for memcached_server
on each role to give the ability for deployers to
override the location of memcached cluster. I.e users
wants to create a single memcached cluster with k8s
for each service.

We also add pymemcache based on [1]

[1] https://review.opendev.org/711429

Change-Id: I64614a02bd3a0d99a69a9c7ac59552ba7cb1fc88
2020-03-16 13:33:53 +00:00
Dmitriy Rabotyagov 229f170c78 Replace git.openstack.org with opendev.org
This patch replaces git.openstack.org with opendev.org as redirection
from old path was enabled.
Also we change upper constraints url due to [1]

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html

Change-Id: I190ba64e727def3202821f943004c87a14a58818
2019-11-14 15:38:11 +02:00
Jonathan Rosser 38bfbf5af5 Allow venv python interpreter to be overridden
Change-Id: I1c062a56edf895287e6dec9a093ed924a146464c
2019-09-11 17:46:34 +01:00
Dmitriy Rabotyagov f3285ed7ad Start using uWSGI role
Move service to use uWSGI role instead of iternal task for uwsgi
deployment. This aims to ease the maintenance of uWSGI and speedup
metal deployments as the same uwsgi environment will be used
across all services.

Change-Id: I5b0bcd6a51f237fcd772368f51a406421ffb52f4
2019-09-05 20:31:19 +03:00
Dmitriy Rabotyagov f0e0919221 Add distribution of metrics.yml
As cloudkitty requires metrics.yml to be present in etc directory
to be fully operational, we're distributing this file by default[1]
cloudkitty_metrics_overrides variable added in order to override default

[1] http://logs.openstack.org/30/673530/2/check/openstack-ansible-functional-ubuntu-bionic/c6eee8a/logs/openstack/openstack1/cloudkitty-processor.service.journal.log.txt.gz#_Jul_30_14_44_00

Change-Id: Ie08cfd0d8dc4835712df0b937170b631aebd10c8
2019-07-31 12:38:56 +03:00
Dmitriy Rabotyagov 6fb1e18a97 Use systemd-journald instead of log files
This patch aims to migrate service from usage of regular syslog files
to journald. We also disable uwsgi logging, since it dublicates
requests that are logged by service itself.

Change-Id: I4cd3270ef873b6f595b151014e82c8b4b35f12aa
2019-07-30 17:16:19 +03:00
Dmitriy Rabotjagov fb6af2f3f0 Update role for new source build process
The variables cloudkitty_developer_mode and cloudkitty_venv_download
no longer carry any meaning. This review changes cloudkitty to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.

As part of this, we move the installation out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.

We also change include_tasks to import_tasks and include_role
to import_role so that the tags in the python_venv_build role
will work.

Change-Id: Ieaec0a12a1ed6ea1f9a98a615fd8c6bff67a9d0c
2019-03-28 15:45:56 +02:00
Jesse Pretorius 12effbdbd3 Enable overriding the service setup host python interpreter
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.

Change-Id: I3f6d4ed50fef2bca61f2bfdc158da804ab80b892
2018-11-30 15:39:56 +00:00
Christian Zunker cdfa3b13c9 Configure cloudkitty-api to use uwsgi
Accoring to the cloudkitty docs it is strongly recommended to
install the API server in a WSGI host such as mod_wsgi.

The tasks are a combination of os_cinder steps and the cloudkitty
docs for wsgi:
https://docs.openstack.org/cloudkitty/latest/install/mod_wsgi.html#mod-wsgi

Change-Id: I4173a32f4989f7e23205083738ccd81f7ea6f9f6
Closes-Bug: 1769632
2018-11-20 11:31:48 +01:00
Jesse Pretorius 374b8c2bce Use a common python build/install role
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.

We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.

This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:

1. Replaces 'developer mode' with an equivalent mechanism
   that uses the common role and is simpler to understand.
   We will also simplify the provisioning of pip install
   arguments when doing this.
2. Simplifies the installation of optional pip packages.
   Right now it's more complicated than it needs to be due
   to us needing to keep the py_pkgs plugin working in the
   integrated build.

Depends-On: https://review.openstack.org/598957
Change-Id: Ife2f3e96f7a56279f68509259c099bfe20ff19d8
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2018-09-03 11:59:33 +00:00
Kevin Carter b78966cd09
Convert role to use a common systemd service role
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed.

The systemd journal would normally be populated with the standard out of
a service however with the use of uwsgi this is not actually happening
resulting in us only capturing the logs from the uwsgi process instead
of the service itself. This change implements journal logging in the
service config, which is part of OSLO logging.

OSLO logging docs found here: <https://docs.openstack.org/oslo.log/3.28.1/journal.html>

Change-Id: Ieff0119cb09e8971fab5b0393f023010a554edff
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-31 14:50:12 -05:00
Jesse Pretorius 1ab9470007 Cleanup playbook, vars and overrides
1. The venv tag override is unnecessary, given there is a default.
2. The install branch override is unnecessary, given there is a
   default. The constraint in defaults was set to use the
   'cloudkitty_requirements_git_install_branch' instead of this
   variable anyway. This has been corrected to the right var.
3. The galera address for role tests has a specific var for it.
4. The 'cloudkitty_requirements_git_install_branch' is a relic
   of the past, and has been removed.
5. The playbook is very dated and is using a lot of old mechanisms
   and unnecessary vars. It is updated to be normalised with the
   playbooks from the integrated build.

Change-Id: I6a8e8b3ec8b00790416648beb1c4ce0257406409
2018-07-31 08:31:52 +01:00
Jesse Pretorius 19f299854e Move MQ vhost/user creation into role
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.

Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.

In this patch we implement two new variables:
- cloudkitty_oslomsg_rpc_setup_host
- cloudkitty_oslomsg_notify_setup_host

These are used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.

We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.

Finally, we remove the test mq setup tasks and clean up any unused
or unnecessary variables configured in tests.

Change-Id: Iac217b80df3da79a944185a90ba4c1ff5409945b
2018-07-31 08:31:19 +01:00
Andy Smith cfddca21eb Update to use oslo.messaging service RPC and Notify
This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters replace
the rabbitmq values and are used to generate the messaging
transport_url for the service.

This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Update transport_url generation in conf
* Add oslo.messaging to tests inventory and update tests
* Install extra packages for optional drivers
* Add release note

Change-Id: I0612252927f81b3584054f6cd77a2826c5d3fc3b
2018-07-31 08:31:18 +01:00
Zuul 4c840ae84c Merge "Update the default collector" 2018-07-11 20:09:02 +00:00
Jesse Pretorius ddb31b12a8 Execute service setup against a delegated host using Ansible built-in modules
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.

The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.

With the dependent patches, the openstack_openrc role is now executed once
on the designated host, so it is no longer required as a meta-dependency for
the role.

Depends-On: https://review.openstack.org/579233
Depends-On: https://review.openstack.org/579959
Change-Id: Ib46638d1cb35e677ecfe789a32bb69a024042d6f
2018-07-06 13:48:47 +00:00
Zuul 71df8fbd44 Merge "Move database creation into role" 2018-07-02 15:48:39 +00:00
Zuul e3b387c9a1 Merge "Add packages required for osprofiler" 2018-06-22 15:13:44 +00:00
Christian Zunker 81991f8621 Use coordination to prevent duplicate pricing
Use MySQL as coordination backend for cloudkitty-processors.
Otherwise prices may get calculated multiple times.

Change-Id: I661ca3b7ffb4f64242422297bafc938ca5082a12
Closes-Bug: 1777432
2018-06-18 14:16:16 +02:00
Jesse Pretorius 3fec5b02ec Move database creation into role
There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.

Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.

In this patch we implement a new variable called 'cloudkitty_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable cloudkitty_galera_address which
has been used for a long time, but never documented.

Change-Id: I7bb3f38556558ef14723361d3672a1d06f839b52
2018-06-12 20:40:05 +01:00
Kevin Carter 610153f5af
Add packages required for osprofiler
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.

Change-Id: Ib058bc3dc6e97871f9c3a177c7f7dec48709ec3f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-11 22:57:34 -05:00
chenxing af1a740da5 Update the default collector
As the ceilometer collector has been removed from cloudkitty repo[1],
we should stop to use it.

[1]https://review.openstack.org/#/c/548630/

Change-Id: I9c244e6a763c2fd82a1f32915e033c4c1b0df280
2018-05-25 18:51:49 +08:00
Zuul ca8e607d86 Merge "Loosen up the sqlalchemy requirements" 2018-05-09 10:24:52 +00:00
Christian Zunker 9b523f9cc9 Use two seperate service variables
Use one variable for collected openstack services and another for
systemd services.

Change-Id: I1946243df319c00638e4ec4a4f1f144fd2131956
Closes-Bug: 1769625
2018-05-07 13:18:53 +02:00
Christian Zunker 8de46d1d7f Loosen up the sqlalchemy requirements
The upstream requierements do not fit openstack-ansible venvs.
So explicitly loosen up until fixed upstream.

Change-Id: I307c9eefac4ab25a88bee7621461e41320bc4a70
Closes-Bug: 1769621
2018-05-07 13:07:47 +02:00
Jean-Philippe Evrard 3fe30720ec Replace virtualenv-tools by a script
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.

Because we can't modify virtualenv-tools, we use shell scripts
instead.

Change-Id: I7a363fa0890101fd2f4007a866ca73b2f0e80d60
Partial-Bug: #1741634
2018-01-15 14:08:30 +00:00
Jenkins 9b64ba6b11 Merge "Deprecate rpc_backend option" 2017-06-08 21:29:51 +00:00
Jesse Pretorius 7aa18adfec Switch to Cryptography over pycrypto
The keystonemiddleware library recently switched to using the
cryptography library over pycrypto, which was unmaintained. See
Iced7f5115e49ccf4f7f5bf6813cb5988b95c248b

Change-Id: I08e9ca9ab28c9f9a81305d5a9865e19f6d0012de
Co-Authored-By: Nolan Brubaker <nolan.brubaker@rackspace.com>
2017-06-07 17:46:01 +00:00
ZhongShengping 1a65676306 Deprecate rpc_backend option
Option "rpc_backend" from group "DEFAULT" is deprecated for removal
(Replaced by [DEFAULT]/transport_url). Its value may be silently
ignored in the future.

Change-Id: Ia352ef7377e6e1f51fffb3181ad5ab422275f39a
Implements: blueprint deprecate-rpc-backend
2017-06-04 22:10:14 +08:00
Kevin Carter bdc17256d2 Ensure the components are isolated from the system
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.

See the following for more information on slices:

* https://www.freedesktop.org/software/systemd/man/systemd.slice.html

See for following for more information on resource controls:

* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html

Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.

Change-Id: Idac3056145b92adcd0b5b328296c59b71bf28441
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2017-03-28 23:34:00 -05:00
Logan V 074ae8281b Clean up developer mode logic
Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server

Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.

Change-Id: I8bb1ff09d676ce3d2faa2baee385695bffb2db59
2017-01-13 19:52:41 -06:00
ZhongShengping 4161ba9381 Remove pki support
Change-Id: Ib579885209f447b501fcb338da4d9321c0aa4498
Implements: blueprint remove-pki
2017-01-04 07:54:48 +00:00
Pedro Perez e3bfd2848f Clean up unused vars
Change-Id: I72b791bdad0ffb7c928eb4c627094968d179173d
2016-11-23 21:10:32 +01:00
Pedro Perez be53f8eca8 update naming conventions for secret vars
Change-Id: I523b5f1ae56c1ff19ca6bd95e3e0e7f00e282c71
2016-11-18 16:42:48 +01:00
Pedro Perez abf382bb68 Add rabbit_port to cloudkitty.conf template
Change-Id: I0dc824ccd4e46d9ab76d0908a1f28f3e9b1fe927
Closes-Bug: #1641695
2016-11-14 21:10:36 +01:00
Jesse Pretorius 0997138c0e Fix linting issues for ansible-lint 3.4.1
Preparing this role for the ansible-lint version bump

Change-Id: I36f5b951b5dcaa52e35079f32e98f2aae94e2df0
2016-11-07 09:57:08 +00:00