This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.
In order to ensure upgrade path and ability to switch back to HA queues
we change vhost names with removing leading `/`, as enabling quorum
requires to remove exchange which is tricky thing to do with running
services.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/896017
Change-Id: I947be8d94b3263ed69311667af693a481765b1c4
Keystone has stopped providing or reffering `_member_` role for a while,
thus role should not be refferenced anymore.
Moreover, with 2023.1 service policies have dropped `_member_`
which resulted in the role to be insufficient for basic operations.
Change-Id: Ie43a6edc4ef44b7b92905cf9d59be53edeb1b946
Related-Bug: #2029486
With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.
With that we also update metdata to reflect current state.
Depends-On: https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/888223
Change-Id: I30eff91392dcab1e76c4fee89ead7a6e03838b2d
By overriding the variable `sahara_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the sahara backend api.
The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: I28e2a2ac7a2534f731f1ce8b0c76f6c55e987eb2
Implement support for service_tokens. For that we convert
role_name to be a list along with renaming corresponding variable.
Additionally service_type is defined now for keystone_authtoken which
enables to validate tokens with restricted access rules
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690
Change-Id: I7064765e62d9e1a86fb20232429731840f697a88
- Implemented new variable ``connection_recycle_time`` responsible for SQLAlchemy's connection recycling
- Set new default values for db pooling variables which are inherited from the global ones.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/819424
Change-Id: I1745cd91c26bb873a5eac0fa42f651d2ebe7e974
With PKI role in place in most cases you don't need to explicitly
provide path to the CA file because PKI role ensures that CA is trusted
by the system overall. In the meanwhile in PyMySQL [1] you must either
provide CA file or cert/key or enable verify.
Since current behaviour is to provide path to the custom CA we expect
certificate being trusted overall. Thus we enable cert verification when
galera_use_ssl is True.
[1] 78f0cf99e5/pymysql/connections.py (L267)
Change-Id: I25b6e2afcf67f34d9b612adca6c0c6968b6308ce
Instead of overriding each service separatelly it might make
sense for deployers to define some higher level variable that
will be used first or fallback to default variable.
Change-Id: I2b9393b388840c8903ca267b8d5e66536be8d267
Even the most modest 4C/8T system would run with the maximum 16 processes
due to the calculation being VCPU*2.
We devide amount of CPUs to number of threads for hyperthreaded CPUs
Change-Id: Ie4384375f65d6ec262a3f5b71ab7cd62ed5e210a
Move it to the service setup host (defaults to utility[0]) instead
of the galera[0] host, and use galera_address (defaults to internal VIP)
as the endpoint instead of a local connection on the db host.
Change-Id: I1564252d2362dbb5858e1d0222c3b344488d389d
This patch aims to add a prefix for memcached_server
on each role to give the ability for deployers to
override the location of memcached cluster. I.e users
wants to create a single memcached cluster with k8s
for each service.
We also add pymemcache based on [1]
[1] https://review.opendev.org/711429
Change-Id: Icfb171b7b5bc33bd6f14378003c6fc9bb597837b
Move service to use uWSGI role instead of iternal task for uwsgi
deployment. This aims to ease the maintenance of uWSGI and speedup
metal deployments as the same uwsgi environment will be used
across all services.
Change-Id: Iaca1e2f680d7281b4d8c0fd47907823a515f4240
This patch aims to migrate service from usage of regular syslog files
to journald. We also disable uwsgi logging, since it dublicates
requests that are logged by service itself.
Change-Id: I0f9eeeb5db890ba2119f7e8a5e85b9f6923092d6
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing
features or functionality. The intention of this change is to ensure
uniformity and reduce the maintenance burden on the community when
sweeping changes are needed. The exterior role is built to be OSA
compatible and may be pulled into tree should we deem it necessary.
Change-Id: If8a201dd964ea769c688f78abc2a688782e3be4d
The variables sahara_developer_mode and sahara_venv_download
no longer carry any meaning. This review changes sahara to do
the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.
As part of this, we move the source build out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.
Change-Id: I74f5fa25b70fbb5c514af3a72b5b6654d5c8e24d
py_pkgs is not yet deprecated, nor does it get parsable by ansible.
We need a static list instead.
This replaces the dynamic list creation with a static list.
Change-Id: Ie2238a4860a110e5fc64e2c55246535467e9077b
- install also the Sahara plugins. The existing variable sahara_plugin_base
is now used also for the list of installed plugins, in addition to
the list of loaded plugins.
The default set includes now all the available plugins.
- fix the URIs used by role tests. They will eventually go away
in favor of the integrated tests, but they are needed right now.
Change-Id: I5a4066ad9e2cca7bbfeb82bca5b2e65badef2e22
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.
Change-Id: I6ff22dd4438df02b4a485ae75e8ea895c0f8f57e
Sahara supports unversioned endpoint also for the stable API v1.1,
and this is a requirement in order to use the experimental API v2.
See https://review.openstack.org/#/c/582285/ for more details.
Following the pattern used for other services with unversioned
endpoints, the sahara_service_*uri variables are directly and all
sahara_service_*url can thus be removed.
Closes-Bug: #1782147
Change-Id: I2d4477de6a3ad58fe58152b18e18a9a6dffdafd4
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.
We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.
This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:
1. Replaces 'developer mode' with an equivalent mechanism
that uses the common role and is simpler to understand.
We will also simplify the provisioning of pip install
arguments when doing this.
Depends-On: https://review.openstack.org/598957
Change-Id: Ibd021f211f4608636e27283ca831aac4e3ef4efe
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
When the RPC and Notify service are the same, the credentials
must match - otherwise the tasks to create the user/password
will overwrite with each other.
If the two clusters are different, then the matching credentials
and vhost will not be a problem. However, if the deployer really
wishes to make sure they're different, then the vars can be
overridden.
Also, to ensure that the SSL value is consistently set in the
conf file, we apply the bool filter. We also use the 'notify'
SSL setting as the messaging system for Notifications is more
likely to remain rabbitmq in our default deployment with qrouterd
becoming the default for RPC messaging.
Change-Id: I0d5285eeebe94adda39b93e61e79be5673b7347d
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement two new variables:
- sahara_oslomsg_rpc_setup_host
- sahara_oslomsg_notify_setup_host
These are used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.
We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.
Change-Id: I72c26ad851beb5a48cd2d841dca67c547a074847
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
Change-Id: Ib4ff0ca76eb7b2ccaa373eeb1812e3d6741193fc
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.
Depends-On: I3df2c670beeb78baaa1515bcd27e8f2b0d95b3a9
Change-Id: I1952353c0eea966a02e5cb20140170d671808b42
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement a new variable called 'sahara_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable sahara_galera_address which
has been used for a long time, but never documented.
Change-Id: I594ef44b2943dc9d7a45771bd1e8592595bd0efc
This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters
replace the rabbitmq values and are used to generate the messaging
transport_url for the service. The association of the messaging
backend server to the oslo.messaging services will then be
transparent to the sahara service.
This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Update transport_url generation
* Add oslo.messaging to inventory
* Add release note
Depends-On: If4326a6848d2d32af284fdbb94798eb0b03734d5
Depends-On: I2b09145b60116c029fc85477399c24f94974b61d
Change-Id: Ib4153412b91e7d11cc9acdbe5af8a1a4280b44e8
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: I5c8b8d102829e84179f94e10a7464008ac5753c4
Partial-Bug: #1741634
When 'sahara_galera_use_ssl' is True, use an encrypted connection to
the database using either a self-signed or user-provided CA certificate.
A new non-voting test has been added to verify that the role remains
functional when enabling SSL features.
Change-Id: Ie39024c99cad0932f464707adc43019bc1062317
Partial-Bug: 1667789
Nova network has been fully removed from the OpenStack codebase.
As such, all instances of switches on use_neutron should be removed,
as the functionality will never be meaningful.
Change-Id: I2eb338ba53d51c62c88bbbfd145462030596da74
Depends-On: Ib9d87dd339d637b69fb27315d92228cbc523c8eb
Closes-Bug: #1734615
As part of the Pike goals we are moving api services to run as WSGI
apps. sahara-api service is set up as a wsgi app, and this patch
moves it over.
Since this is just a drop in replacement for the existing eventlet
service, operators an deployers should notice no difference.
Change-Id: Ie4826358d5ee1686ad9ea7fa9eb0441acff565f2
Implements: blueprint goal-deploy-api-in-wsgi
The keystonemiddleware library recently switched to using the
cryptography library over pycrypto, which was unmaintained. See
Iced7f5115e49ccf4f7f5bf6813cb5988b95c248b
Change-Id: I8a6e828664d4c847e707c9fb5b18cfc99917b4a3
Co-Authored-By: Nolan Brubaker <nolan.brubaker@rackspace.com>
Option "rpc_backend" from group "DEFAULT" is deprecated for removal
(Replaced by [DEFAULT]/transport_url). Its value may be silently
ignored in the future.
Change-Id: I6ff5a9e734d5ba3de12a43e6f57d5a3ce214db5e
Implements: blueprint deprecate-rpc-backend
This creates a specific slice which all OpenStack services will operate
from. By creating an independent slice these components will be governed
away from the system slice allowing us to better optimise resource
consumption.
See the following for more information on slices:
* https://www.freedesktop.org/software/systemd/man/systemd.slice.html
See for following for more information on resource controls:
* https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
Tools like ``systemd-cgtop`` and ``systemd-cgls`` will now give us
insight into specific processes, process groups, and resouce consumption
in ways that we've not had access to before. To enable some of this reporting
the accounting options have been added to the [Service] section of the unit
file.
Change-Id: Ic4edfa39b8ee42d0f6192b986e1c40c9c94488df
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Users can configure the number of worker threads however when it's
not specified the calculated number of workers can get too large on
hosts with a large number of CPUs.
Also removed the comment related to an unused variable
sahara_engine_workers in defaults/main.yml
Change-Id: Idc39c7fc2188332fbaeb771906c0b9a402d17646
- Use dictionary for service group mappings, bringing the
role into line with the method used in other roles;
- Use systemd module instead of shell on the services
restart handlers;
- Use ansible package module to install distro packages;
- Added variables for CentOS. The role should now support
CentOS;
- Removed extras folder.
Change-Id: I5c8430804aacceca01c5821ca2528514033d15f4
Dmitriy Rabotyagov