Commit Graph

83 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov ea5e1adf63 Add quorum support for service
This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.

Change-Id: Ia51971c077cef647c3d4e07d6cbc14b7bac70788
2023-09-28 14:25:08 +00:00
Dmitriy Rabotyagov 1f95cd900e Fix linters and metadata
With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.

With that we also update metdata to reflect current state.

Change-Id: I4585a4aad7acf48080e6b2d73bf3e0c2e0dfbff7
2023-07-14 05:49:48 +00:00
Damian Dabrowski 4f785b4e5f Add TLS support to barbican backends
By overriding the variable `barbican_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the barbican backend api.

The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: I68abd8b2c63231ece3b7184d6e52168cee5ce3d1
2023-04-29 18:35:45 +02:00
Dmitriy Rabotyagov cb6c38ab92 Support service tokens
Implement support for service_tokens. For that we convert
role_name to be a list along with renaming corresponding variable.

Additionally service_type is defined now for keystone_authtoken which
enables to validate tokens with restricted access rules

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690
Change-Id: I6a5e16a4fc2a81dedc4bc459f13ac7781292f5a8
2022-06-15 17:40:02 +02:00
Andrew Bonney 38c3964255 Remove SSL variables which appear to be unused
Whilst enabling TLS v1.3 in other roles these variables were noted
which don't appear to be used anywhere in the role.

Change-Id: I6b06486328ec0af05a17272be99a14911be9f4f7
2022-01-10 10:54:29 +00:00
Damian Dabrowski 3e642f2f72 Database connection pooling improvements
- Implemented new variable ``connection_recycle_time`` responsible for SQLAlchemy's connection recycling
- Set new default values for db pooling variables which are inherited from the global ones.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/819424
Change-Id: Ic58f085c8b1250b1db831fa8c74215abd2519704
2021-12-03 11:39:55 +01:00
Dmitriy Rabotyagov 13042f76c3 Refactor galera_use_ssl behaviour
With PKI role in place in most cases you don't need to explicitly
provide path to the CA file because PKI role ensures that CA is trusted
by the system overall. In the meanwhile in PyMySQL [1] you must either
provide CA file or cert/key or enable verify.

Since current behaviour is to provide path to the custom CA we expect
certificate being trusted overall. Thus we enable cert verification when
galera_use_ssl is True.

[1] 78f0cf99e5/pymysql/connections.py (L267)

Change-Id: Ia55488a9fcc6b2824276bf824197ae8ea7af0177
2021-09-17 17:00:33 +03:00
Dmitriy Rabotyagov 05c5ab38e4 Allow to symlink barbican_user_libraries
When barbican uses PKCS#11 crypt plugin, libCryptoki2 library tends to
search for Chrystoki.conf inside /etc by default. At the same time it's
tricky to place file there at once since approriate permissions not
always could be set for files that reside directly in /etc.
As a workaround to this Chrystoki.conf can be placed inside /opt and
symlinked to /etc to satisfy library.

Change-Id: I6267d3b65f514c4ad4cb5494f111463e685b6fbb
2021-06-24 20:05:51 +03:00
Jonathan Rosser 55595fa93e Add variables for rabbitmq ssl configuration
Change-Id: I69b5d844e5fd20bc8078910f27999f4ece66f47f
2021-05-17 07:55:24 +00:00
Jonathan Rosser 5f62076c0e Use ansible_facts[] instead of fact variables
See https://github.com/ansible/ansible/issues/73654

Change-Id: I45f7032da03ae5b74924b8d1355ed3f72a0830aa
2021-03-16 07:59:45 +00:00
Jonathan Rosser fe9a859de5 Switch default virtualenv to python3
Change-Id: I3e52273517a8857446a0446aa8229ce86c21d261
2021-03-10 08:42:00 +00:00
Jonathan Rosser 78be497957 Move barbican pip packages from constraints to requirements
This is necessary to support the new pip resolver

Change-Id: I2ce8e7494dc367bd64fb36947ebce0e7c949c493
2021-01-25 08:48:19 +00:00
Dmitriy Rabotyagov f43f76e27a Use global service variables
Instead of overriding each service separatelly it might make
sense for deployers to define some higher level variable that
will be used first or fallback to default variable.

Change-Id: Idea74ec31c3a5a9edbf9f32b81563256e0d496b9
2021-01-05 18:34:35 +02:00
Zuul ddf65cdc3e Merge "Add deployment of the external libraries" 2020-11-27 19:59:35 +00:00
Zuul 34c28b79f9 Merge "Allow multibackend support for Barbican" 2020-11-27 19:59:31 +00:00
Zuul 4b546feb6a Merge "Clean up barbican.conf" 2020-11-27 19:59:27 +00:00
Zuul 2d47f6ba2e Merge "Cleanup stop handler and barbican_apache_* variables" 2020-11-24 13:37:56 +00:00
Dmitriy Rabotyagov 3acae8f8c9 Add deployment of the external libraries
Deployment of user libraries might be needed for interaction of PKCS#11 module
with external HSM solutions.

Change-Id: I0a0754052a0d48792322243341171593bbbd1a41
2020-11-16 15:59:21 +02:00
Dmitriy Rabotyagov 50c983e034 Allow multibackend support for Barbican
This patch introduces 2 new variables that are designed to help deployer
with barbican configuration. They are designed to support multibackend
caonfiguration of the barbican while default behavior should not change.

Change-Id: I3369c4254f3b48f12ed9731f18d980044e6d0b43
2020-11-14 07:29:30 +00:00
Dmitriy Rabotyagov 76b72c0975 Clean up barbican.conf
Drop out default or misconfigured variables from barbican.conf to
make config file readable.
This should not affect existing deployments since plugin config has to be
overriden anyway.

Depends-On: https://review.opendev.org/759082
Change-Id: I2a0756b851c9e862b2312b47d37b723386d6915c
2020-11-13 20:34:55 +00:00
Dmitriy Rabotyagov ab1b2ac476 Reduce number of processes on small systems
Even the most modest 4C/8T system would run with the maximum 16 processes
due to the calculation being VCPU*2.

We devide amount of CPUs to number of threads for hyperthreaded CPUs

Change-Id: I1181959604a59d0f599f5a0f0a43e348649ba74f
2020-11-13 18:03:12 +00:00
Dmitriy Rabotyagov 8906fac35d Cleanup stop handler and barbican_apache_* variables
Since handler was added for upgrade purposes, we can drop it now.
We also remove not used barbican_apache_* variables since we've migrated
to the uwsgi usage several releases ago.

Change-Id: Ib19834a1ae4751f7439afaabb532220f873e4b8f
2020-11-13 16:32:42 +00:00
Dmitriy Rabotyagov 3840c883c4 Use the utility host for db setup tasks
Move it to the service setup host (defaults to utility[0]) instead
of the galera[0] host, and use galera_address (defaults to internal VIP)
as the endpoint instead of a local connection on the db host.

Change-Id: I11c5b7a79e3a5b37fd5cef5aa46a1c0a6c654c9d
2020-08-20 17:14:50 +03:00
Dmitriy Rabotyagov 5de6e8084f Cleanup after repo_build and pip_install retirement
Change-Id: I8ece554f5a2bb91074b9bc4dcc3d84816387f07c
2020-05-12 21:11:16 +03:00
Guilherme Steinmüller 564c3e8935 Refactor memcached_servers
This patch aims to add a prefix for memcached_server
on each role to give the ability for deployers to
override the location of memcached cluster. I.e users
wants to create a single memcached cluster with k8s
for each service.

We also add pymemcache based on [1]

[1] https://review.opendev.org/711429

Change-Id: Ibd5decc06f205f5e1de9dbc0d7e9cde5e9435c4e
2020-03-16 13:27:35 +00:00
Dmitriy Rabotyagov fdc249db05 Replace git.openstack.org with opendev.org
This patch replaces git.openstack.org with opendev.org as redirection
from old path was enabled.
Also we change upper constraints url due to [1]

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html

Depends-On: https://review.opendev.org/693841/
Change-Id: Ibeef50ce58bf793192477ee939ef136ea3d053e6
2019-11-14 16:21:37 +00:00
Jonathan Rosser 16ac9c2a5a Add global override for service bind address
Change-Id: Ia929974da1035406c47d609bd1bca102a79574ed
2019-09-19 09:48:17 +01:00
Jonathan Rosser c00363e3d7 Allow venv python interpreter to be overridden
Change-Id: Ic88b8faa6ee52db263d36d255d03da919405dab6
2019-09-11 17:42:43 +01:00
Zuul db3088a256 Merge "Add extra barbican services" 2019-09-10 05:23:04 +00:00
Dmitriy Rabotyagov 51c0311478 Start using uWSGI role
Move service to use uWSGI role instead of iternal task for uwsgi
deployment. This aims to ease the maintenance of uWSGI and speedup
metal deployments as the same uwsgi environment will be used
across all services.

Change-Id: I8492d52d8155e1bd75f99deb4fd19d445a831816
2019-09-04 16:39:05 +03:00
Dmitriy Rabotyagov 434eebd0c3 Add extra barbican services
This patch adds option to easily enable barbican extra services.
It replaces barbican service with barbican-api with same functionality.

Change-Id: I76494288e17f1801b4f1055cbb4d9b6b4bd4f475
2019-09-04 13:06:32 +03:00
Mohammed Naser ea70314063 Update role for new source build process
The variables barbican_developer_mode and barbican_venv_download
no longer carry any meaning. This review changes glance to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.

As part of this, we move the source build out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.

Change-Id: I58d0dc682f0e061f003b022c46277a4849ba4b2b
2019-03-26 14:49:35 -04:00
Zuul d421fe7381 Merge "Force force-tlsv12 only" 2019-01-18 16:50:29 +00:00
Matthew Thode 84435d5bc0
Force force-tlsv12 only
Secure by default

Depends-On: https://review.openstack.org/628979
Change-Id: I068876e37bb7c51dac074f98c8d163237dfe44e1
2019-01-09 10:43:25 -06:00
Jesse Pretorius 86c8f7fee2 Enable overriding the service setup host python interpreter
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.

Change-Id: Id3727087b8302258875aa175b2bd2b2a6b4ebe0e
2018-11-30 15:31:27 +00:00
Andy Smith de2cbdccdb Add install packages for optional oslo.messaging amqp1 driver
Change-Id: I12f4de8325a525da614f6678f22ee950d4fe54bd
2018-09-11 10:02:16 -04:00
Jesse Pretorius a3d934f9e5 Use a common python build/install role
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.

We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.

This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:

1. Replaces 'developer mode' with an equivalent mechanism
   that uses the common role and is simpler to understand.
   We will also simplify the provisioning of pip install
   arguments when doing this.
2. Simplifies the installation of optional pip packages.
   Right now it's more complicated than it needs to be due
   to us needing to keep the py_pkgs plugin working in the
   integrated build.
3. Deduplicates the distro package installs. Right now the
   role installs the distro packages twice - just before
   building the venv, and during the python_venv_build role
   execution.

Depends-On: https://review.openstack.org/598957
Change-Id: I1b07bbdfc5ddbc42b40daf00da5ebbc7502eeccf
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2018-09-03 11:05:53 +00:00
Zuul e955881385 Merge "Setup oslo.messaging extra packages for optional drivers" 2018-08-09 06:23:00 +00:00
Kevin Carter 005e5b61ca
Convert role to use a common systemd service role
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed.

The systemd journal would normally be populated with the standard out of
a service however with the use of uwsgi this is not actually happening
resulting in us only capturing the logs from the uwsgi process instead
of the service itself. This change implements journal logging in the
service config, which is part of OSLO logging.

OSLO logging docs found here: <https://docs.openstack.org/oslo.log/3.28.1/journal.html>

Change-Id: Ic5b57a650bd9f5c385ed0a0a3efd1d530a2d7e81
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-31 14:43:44 -05:00
Andy Smith 2f7002ff9d Setup oslo.messaging extra packages for optional drivers
Change-Id: I6166fc192de6450b6e524077c94273ecece2d30d
2018-07-27 16:06:41 -04:00
Jesse Pretorius 8557ee5bc5 Move MQ vhost/user creation into role
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.

Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.

In this patch we implement two new variables:
- barbican_oslomsg_rpc_setup_host
- barbican_oslomsg_notify_setup_host

These are used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.

We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.

Finally, we remove the test mq setup tasks and clean up any unused
or unnecessary variables configured in tests.

Change-Id: I8e873ad9dc8184ec387fe5e1bd95af1b90bb7233
2018-07-27 12:50:48 +01:00
Andrew Smith 206f411451 Update to use oslo.messaging service for RPC and Notify
This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters replace
the rabbitmq values and are used to generate the messaging
transport_url for the service. The association of the messaging
backend server to the oslo.messaging services will then be transparent
to the barbican service.

This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Add transport_url generation to conf
* Add oslo.messaging to tests inventory
* Update tets
* Update examples
* Add release note

Change-Id: I0657c88799e06987c6df90edd55fda859faf6035
2018-07-26 09:37:01 +00:00
Zuul e4b09bf175 Merge "Add packages required for osprofiler" 2018-07-18 22:23:10 +00:00
Vadim Kuznetsov 0ed71b4f48 Add default roles used by Barbican
The default roles used by Barbican are missing.
According to the policy.json the Key Manager
comes with, four roles have to be defined.

Depends-On: I999aa55269348497154a8eb073c2ff663242af4e
Change-Id: If50e02a5886a1f3b26eb033103904122302afeda
Closes-Bug: #1657742
2018-07-12 13:03:18 -04:00
Kevin Carter 2c3cd332c2 Add packages required for osprofiler
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.

Change-Id: I22bf333e42f4961a89eb593c95838fd65a966d93
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-11 04:54:28 +00:00
Jesse Pretorius 9d01669487 Execute service setup against a delegated host using Ansible built-in modules
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.

The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.

With the dependent patches, the openstack_openrc role is now executed once
on the designated host, so it is no longer required as a meta-dependency for
the role.

Depends-On: https://review.openstack.org/579233
Depends-On: https://review.openstack.org/579959
Change-Id: I20186fd3bb4a3fade91e14f6360232454de9c989
2018-07-06 13:49:14 +00:00
Jesse Pretorius 33d5e31717 Move database creation into role
There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.

Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.

In this patch we implement a new variable called 'barbican_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable barbican_galera_address which
has been used for a long time, but never documented.

Change-Id: I6257a172cf52e61cb8219f3698c17821270c3ffb
2018-06-25 09:36:49 +00:00
Markos Chandras c9135f8b34 Add support for using distribution packages for OpenStack services
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones.

Change-Id: I1c2b4ad14fb40ce3958ed197115ccf45468544c6
Implements: blueprint openstack-distribution-packages
2018-05-15 11:42:58 +01:00
Mohammed Naser 4b0da513d0 Allow ability to configure number of processes and threads
Users can configure the number of worker threads however when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs.

This also adds the setting of threads and processes to the UWSGI
configuration.

Change-Id: I003ab426488966cce46bd6fd297c79ada13c9668
Closes-Bug: #1745631
2018-01-28 17:44:43 +00:00
Zuul 5730d33706 Merge "Replace virtualenv-tools by a script" 2018-01-16 12:58:22 +00:00