This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.
In order to ensure upgrade path and ability to switch back to HA queues
we change vhost names with removing leading `/`, as enabling quorum
requires to remove exchange which is tricky thing to do with running
services.
Change-Id: I722aefb5290a68311faf1d44c279151e8492466a
With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.
With that we also update metdata to reflect current state.
Depends-On: https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/888223
Change-Id: Ib51c7ec8d83ffe82c98d1c8d3200ce4510775895
By overriding the variable `cloudkitty_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the cloudkitty backend api.
The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: I0566028b4737306954b544d040b52cee1fddaa1f
Implement support for service_tokens. For that we convert
role_name to be a list along with renaming corresponding variable.
Additionally service_type is defined now for keystone_authtoken which
enables to validate tokens with restricted access rules
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690
Change-Id: Icb1de8c7e0a5196a4df457a5d4a3ca524d4622d0
- Implemented new variable ``connection_recycle_time`` responsible for SQLAlchemy's connection recycling
- Set new default values for db pooling variables which are inherited from the global ones.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/819424
Change-Id: I4ae28b356a404c1e76e5cff986d4fe0172b564f2
With PKI role in place in most cases you don't need to explicitly
provide path to the CA file because PKI role ensures that CA is trusted
by the system overall. In the meanwhile in PyMySQL [1] you must either
provide CA file or cert/key or enable verify.
Since current behaviour is to provide path to the custom CA we expect
certificate being trusted overall. Thus we enable cert verification when
galera_use_ssl is True.
[1] 78f0cf99e5/pymysql/connections.py (L267)
Change-Id: I756a6cdaa4b619671da73c65761c796a0b2ade81
The Cloudkitty role has been lacking some attention for a while
causing some of the configuration and examples outdated.
Now that Cloudkitty playbook and dashboard can be installed using
OSA without running any additional playbooks from this repository
i have removed those from here.
Change-Id: I0fe96c318273f0016d93cf043bda74feb11c63f8
Even the most modest 4C/8T system would run with the maximum 16 processes
due to the calculation being VCPU*2.
We devide amount of CPUs to number of threads for hyperthreaded CPUs
Change-Id: Ifa09cdddcb4a3875f064ff773bbe2281f964fb1d
Move it to the service setup host (defaults to utility[0]) instead
of the galera[0] host, and use galera_address (defaults to internal VIP)
as the endpoint instead of a local connection on the db host.
Change-Id: Ie8e6b95f531fef9c9fe04d6d8f692d599aa5ff30
This patch aims to add a prefix for memcached_server
on each role to give the ability for deployers to
override the location of memcached cluster. I.e users
wants to create a single memcached cluster with k8s
for each service.
We also add pymemcache based on [1]
[1] https://review.opendev.org/711429
Change-Id: I64614a02bd3a0d99a69a9c7ac59552ba7cb1fc88
Move service to use uWSGI role instead of iternal task for uwsgi
deployment. This aims to ease the maintenance of uWSGI and speedup
metal deployments as the same uwsgi environment will be used
across all services.
Change-Id: I5b0bcd6a51f237fcd772368f51a406421ffb52f4
This patch aims to migrate service from usage of regular syslog files
to journald. We also disable uwsgi logging, since it dublicates
requests that are logged by service itself.
Change-Id: I4cd3270ef873b6f595b151014e82c8b4b35f12aa
The variables cloudkitty_developer_mode and cloudkitty_venv_download
no longer carry any meaning. This review changes cloudkitty to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.
As part of this, we move the installation out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.
We also change include_tasks to import_tasks and include_role
to import_role so that the tags in the python_venv_build role
will work.
Change-Id: Ieaec0a12a1ed6ea1f9a98a615fd8c6bff67a9d0c
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.
Change-Id: I3f6d4ed50fef2bca61f2bfdc158da804ab80b892
Accoring to the cloudkitty docs it is strongly recommended to
install the API server in a WSGI host such as mod_wsgi.
The tasks are a combination of os_cinder steps and the cloudkitty
docs for wsgi:
https://docs.openstack.org/cloudkitty/latest/install/mod_wsgi.html#mod-wsgi
Change-Id: I4173a32f4989f7e23205083738ccd81f7ea6f9f6
Closes-Bug: 1769632
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.
We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.
This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:
1. Replaces 'developer mode' with an equivalent mechanism
that uses the common role and is simpler to understand.
We will also simplify the provisioning of pip install
arguments when doing this.
2. Simplifies the installation of optional pip packages.
Right now it's more complicated than it needs to be due
to us needing to keep the py_pkgs plugin working in the
integrated build.
Depends-On: https://review.openstack.org/598957
Change-Id: Ife2f3e96f7a56279f68509259c099bfe20ff19d8
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed.
The systemd journal would normally be populated with the standard out of
a service however with the use of uwsgi this is not actually happening
resulting in us only capturing the logs from the uwsgi process instead
of the service itself. This change implements journal logging in the
service config, which is part of OSLO logging.
OSLO logging docs found here: <https://docs.openstack.org/oslo.log/3.28.1/journal.html>
Change-Id: Ieff0119cb09e8971fab5b0393f023010a554edff
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
1. The venv tag override is unnecessary, given there is a default.
2. The install branch override is unnecessary, given there is a
default. The constraint in defaults was set to use the
'cloudkitty_requirements_git_install_branch' instead of this
variable anyway. This has been corrected to the right var.
3. The galera address for role tests has a specific var for it.
4. The 'cloudkitty_requirements_git_install_branch' is a relic
of the past, and has been removed.
5. The playbook is very dated and is using a lot of old mechanisms
and unnecessary vars. It is updated to be normalised with the
playbooks from the integrated build.
Change-Id: I6a8e8b3ec8b00790416648beb1c4ce0257406409
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement two new variables:
- cloudkitty_oslomsg_rpc_setup_host
- cloudkitty_oslomsg_notify_setup_host
These are used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.
We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.
Finally, we remove the test mq setup tasks and clean up any unused
or unnecessary variables configured in tests.
Change-Id: Iac217b80df3da79a944185a90ba4c1ff5409945b
This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters replace
the rabbitmq values and are used to generate the messaging
transport_url for the service.
This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Update transport_url generation in conf
* Add oslo.messaging to tests inventory and update tests
* Install extra packages for optional drivers
* Add release note
Change-Id: I0612252927f81b3584054f6cd77a2826c5d3fc3b
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
With the dependent patches, the openstack_openrc role is now executed once
on the designated host, so it is no longer required as a meta-dependency for
the role.
Depends-On: https://review.openstack.org/579233
Depends-On: https://review.openstack.org/579959
Change-Id: Ib46638d1cb35e677ecfe789a32bb69a024042d6f
Use MySQL as coordination backend for cloudkitty-processors.
Otherwise prices may get calculated multiple times.
Change-Id: I661ca3b7ffb4f64242422297bafc938ca5082a12
Closes-Bug: 1777432
There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement a new variable called 'cloudkitty_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable cloudkitty_galera_address which
has been used for a long time, but never documented.
Change-Id: I7bb3f38556558ef14723361d3672a1d06f839b52
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.
Change-Id: Ib058bc3dc6e97871f9c3a177c7f7dec48709ec3f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
As the ceilometer collector has been removed from cloudkitty repo[1],
we should stop to use it.
[1]https://review.openstack.org/#/c/548630/
Change-Id: I9c244e6a763c2fd82a1f32915e033c4c1b0df280
Use one variable for collected openstack services and another for
systemd services.
Change-Id: I1946243df319c00638e4ec4a4f1f144fd2131956
Closes-Bug: 1769625
The upstream requierements do not fit openstack-ansible venvs.
So explicitly loosen up until fixed upstream.
Change-Id: I307c9eefac4ab25a88bee7621461e41320bc4a70
Closes-Bug: 1769621
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: I7a363fa0890101fd2f4007a866ca73b2f0e80d60
Partial-Bug: #1741634
The keystonemiddleware library recently switched to using the
cryptography library over pycrypto, which was unmaintained. See
Iced7f5115e49ccf4f7f5bf6813cb5988b95c248b
Change-Id: I08e9ca9ab28c9f9a81305d5a9865e19f6d0012de
Co-Authored-By: Nolan Brubaker <nolan.brubaker@rackspace.com>
Option "rpc_backend" from group "DEFAULT" is deprecated for removal
(Replaced by [DEFAULT]/transport_url). Its value may be silently
ignored in the future.
Change-Id: Ia352ef7377e6e1f51fffb3181ad5ab422275f39a
Implements: blueprint deprecate-rpc-backend
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Idac3056145b92adcd0b5b328296c59b71bf28441
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Fixes the ability to deploy a venv in cases where:
1) developer_mode is not enabled
2) A cached venv is not downloaded from the repo server
Additional cleanup to the developer_mode venv deployment
logic is implemented by adding a *_venv_download var
which is used to decouple developer_mode from the
cached venv extraction process so that a deployer
can force venv builds in-place (disable cached
venv usage) without enabling developer mode
constraints.
Change-Id: I8bb1ff09d676ce3d2faa2baee385695bffb2db59