This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.
In order to ensure upgrade path and ability to switch back to HA queues
we change vhost names with removing leading `/`, as enabling quorum
requires to remove exchange which is tricky thing to do with running
services.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/896017
Change-Id: I7e4e8b3be33536545b5b4bcfb4855e8c160bb152
With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.
With that we also update metdata to reflect current state.
Depends-On: https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/888223
Change-Id: I68a3041edf0b0eb891fbe1e40081f779fc40c21d
By overriding the variable `heat_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the heat backend api.
The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: Ifb904adc61f1461e646c3fce0bd062f526b8e446
Implement support for service_tokens. For that we convert
role_name to be a list along with renaming corresponding variable.
Additionally service_type is defined now for keystone_authtoken which
enables to validate tokens with restricted access rules
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690
Change-Id: Ib5d15aaf56112a776e2b9abb2396f9ea4f4fe319
- Implemented new variable ``connection_recycle_time`` responsible for SQLAlchemy's connection recycling
- Set new default values for db pooling variables which are inherited from the global ones.
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/819424
Change-Id: I78301a9d98854ba9f80cf6613e62a363f8327dfc
With PKI role in place in most cases you don't need to explicitly
provide path to the CA file because PKI role ensures that CA is trusted
by the system overall. In the meanwhile in PyMySQL [1] you must either
provide CA file or cert/key or enable verify.
Since current behaviour is to provide path to the custom CA we expect
certificate being trusted overall. Thus we enable cert verification when
galera_use_ssl is True.
[1] 78f0cf99e5/pymysql/connections.py (L267)
Change-Id: I8e689330b76e72df780be3b2f8af066a5fe96a2a
Instead of overriding each service separatelly it might make
sense for deployers to define some higher level variable that
will be used first or fallback to default variable.
Change-Id: I73d5e7334c076f4d0dd18b157307aa31cfb541a9
Even the most modest 4C/8T system would run with the maximum 16 processes
due to the calculation being VCPU*2.
We devide amount of CPUs to number of threads for hyperthreaded CPUs
Change-Id: Ia0304a9314029b5678a3f730d662be6580b17a77
Move it to the service setup host (defaults to utility[0]) instead
of the galera[0] host, and use galera_address (defaults to internal VIP)
as the endpoint instead of a local connection on the db host.
Change-Id: Ic51187c5f063d7d8b932e76d0f82c062061a3962
This patch aims to add a prefix for memcached_server
on each role to give the ability for deployers to
override the location of memcached cluster. I.e users
wants to create a single memcached cluster with k8s
for each service.
We also add pymemcache based on [1]
[1] https://review.opendev.org/711429
Change-Id: Id853b46de4323b605ff6544b1e48bae30a0bf0a7
Move service to use uWSGI role instead of iternal task for uwsgi
deployment. This aims to ease the maintenance of uWSGI and speedup
metal deployments as the same uwsgi environment will be used
across all services.
Change-Id: If539762d5de0730ce97d8f8aaefb4c096d850902
This patch aims to migrate service from usage of regular syslog files
to journald. We also disable uwsgi logging, since it dublicates
requests that are logged by service itself.
Change-Id: I9f8a7f92d644775bd673dbade73fbaa1268712d5
The variables heat_developer_mode and heat_venv_download
no longer carry any meaning. This review changes heat to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.
As part of this, we move the installation out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.
Depends-On: https://review.openstack.org/648551
Change-Id: I4b66febefc77176a112486b0e9ee9b46e16cff05
With this variable, users would be able to extend
the list of pip packages in case of needing an
extra pip package.
Currently if we need an extra pip package we need
to override the existing list.
Change-Id: I9aea9f42c476ff3c6f2355a0afb21be4eea57b69
https://review.openstack.org/606160 changes the change the dist name
to 'openstack-heat' and it needs to be fixed in the os_heat role
otherwise it will fails while installing it.
Added openmpi-devel distro packages as it is need for building mpi4py.
Depends-On: https://review.openstack.org/636297
Change-Id: I2fbb2465f9b4765a87011dfb2c2f65bd27e7b2c9
The files and templates we carry are almost always in a state of
maintenance. The upstream services are maintaining these files and
there's really no reason we need to carry duplicate copies of them. This
change removes all of the files we expect to get from the upstream
service. while the focus of this change is to remove configuration file
maintenance burdens it also allows the role to execute faster.
* Source installs have the configuration files within the venv at
"<<VENV_PATH>>/etc/<<SERVICE_NAME>>". The role will now link the
default configuration path to this directory. When the service is
upgraded the link will move to the new venv path.
* Distro installs package all of the required configuration files.
To maintain our current capabilities to override configuration the
role will fetch files from the disk whenever an override is provided and
then push the fetched file back to the target using `config_template`.
Change-Id: Ib26b25c1df483c2e31bccc91874a83292be6a3aa
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.
Change-Id: I3bc4b345ede2dff088bd8067c04b1a4cc51529bd
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.
We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.
This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:
1. Replaces 'developer mode' with an equivalent mechanism
that uses the common role and is simpler to understand.
We will also simplify the provisioning of pip install
arguments when doing this.
2. Simplifies the installation of optional pip packages.
Right now it's more complicated than it needs to be due
to us needing to keep the py_pkgs plugin working in the
integrated build.
3. Deduplicates the distro package installs. Right now the
role installs the distro packages twice - just before
building the venv, and during the python_venv_build role
execution.
Depends-On: https://review.openstack.org/598957
Change-Id: I7660ab1cff41a74eca84b8e3fbc1a8f6f1af032b
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
When the RPC and Notify service are the same, the credentials
must match - otherwise the tasks to create the user/password
will overwrite with each other.
If the two clusters are different, then the matching credentials
and vhost will not be a problem. However, if the deployer really
wishes to make sure they're different, then the vars can be
overridden.
Also, to ensure that the SSL value is consistently set in the
conf file, we apply the bool filter. We also use the 'notify'
SSL setting as the messaging system for Notifications is more
likely to remain rabbitmq in our default deployment with qrouterd
becoming the default for RPC messaging.
Change-Id: I9aa597705514a460a6bc57137b7d2d06bf6b7f05
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement two new variables:
- heat_oslomsg_rpc_setup_host
- heat_oslomsg_notify_setup_host
These are used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.
We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.
Change-Id: I7deae4d8b8b0e2a7e5c810312a5859900a63901a
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.
The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.
With the dependent patches, the openstack_openrc role is now executed once
on the designated host, so it is no longer required as a meta-dependency for
the role.
Depends-On: https://review.openstack.org/579233
Depends-On: https://review.openstack.org/579959
Depends-On: https://review.openstack.org/580156
Change-Id: I261a49e374d34d00d0be1863da701b7a13ceb3c6
We've switched everything to python-openstackclient and therefore
the usage of python-cinderclient is no longer necessary.
Change-Id: Ic9a276a3ad3d0f0d2ce34803d1d5076b95680c5a
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.
Change-Id: I2719ea6fcb2532fec771ec2316f2d761c9ceccad
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
There is no record for why we implement the database creation outside
of the role in the playbook, when we could do it inside the role.
Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.
In this patch we implement a new variable called 'heat_db_setup_host'
which is used in the role to allow delegation of the database setup
task to any host, but defaults to the first member of the galera_all
host group. We also document the variable heat_galera_address which
has been used for a long time, but never documented.
Change-Id: I5f3e73af031aad455dbd8555aa054eac804a55bb
The keystoneclient package is being installed on the host by PIP but
that means that a whole bunch of required dependencies are being pulled
in as well.
This brings the host to a rather messed up state when installing
keystone from distro packages, since distribution and
PIP packages are being mixed together. We only need the client to
register the service with keystone so we can simply use the distro
package for that to avoid installing lots of PIP packages on the
host.
Implements: blueprint openstack-distribution-packages
Change-Id: I3759995cfb8192145d1b871158c95f594751a759
This introduces oslo.messaging variables that define the RPC and
Notify transports for the OpenStack services. These parameters
replace the rabbitmq values and are used to generate the messaging
transport_url for the service. The association of the messaging
backend server to the oslo.messaging services will then be
transparent to the heat service.
This patch:
* Add oslo.messaging variables for RPC and Notify to defaults
* Update transport_url generation
* Add oslo.messaging to inventory
* Add release note
Depends-On: If4326a6848d2d32af284fdbb94798eb0b03734d5
Depends-On: I2b09145b60116c029fc85477399c24f94974b61d
Change-Id: I4ee0a94028e7390b319d4e1ceb4ab7f7c28ad9ba
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones.
Change-Id: I2435793b73e0a30131b4710b6e7d29dacbb7e6fa
Implements: blueprint openstack-distribution-packages
Distributions normally package uwsgi in a modular fashion so we need
to be use the 'autoload' option to allow it to load all the required
plugins.
Change-Id: I11750219ef70549b1769bede406d6bdda097b6c2
Implements: blueprint openstack-distribution-packages
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.
Change-Id: Ic46d7f9615a492a98abecc970b72413b727290ac
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.
Because we can't modify virtualenv-tools, we use shell scripts
instead.
Change-Id: Ic2497f23d0c0a9160fc3fe57ccdb78094d499f83
Partial-Bug: #1741634
When 'heat_galera_use_ssl' is True, use an encrypted connection to
the database using either a self-signed or user-provided CA certificate.
A new non-voting test has been added to verify that the role remains
functional when enabling SSL features.
Change-Id: I8d582a5fe18b33592461f3ea23551c97720b8e7d
Partial-Bug: 1667789
As part of the Pike goals we are moving api services to run as WSGI
apps. heat-api, heat-api-cfn, and heat-api-cloudwatch are now set
up as WSGI apps.
Since this is just a drop in replacement for existing eventlet services,
operators and deployers should not notice a difference.
Change-Id: I3dba17c33a7f1a1b9a03020a650e258099b4d20d
Implements: blueprint goal-deploy-api-in-wsgi
In order to optimise the role execution and
to improve consistency, the following is done:
1. The services dict is filtered per host to
prevent skipped tasks from spamming the
logs and wasting execution time.
2. The service restart is implemented in a
specific order to improve consistency. The
order implemented is a wild guess as there
is no upstream documentation about the
'correct' order.
3. The service enable/start is consolidated
into the handler to be more efficient.
4. A dynamic include is used instead of nesting
includes for the init prep.
Change-Id: I91d20b980f1b1a63ff2d6ef30bcec7ef58a5571a
Option "rpc_backend" from group "DEFAULT" is deprecated for removal
(Replaced by [DEFAULT]/transport_url). Its value may be silently
ignored in the future.
Change-Id: I77bb1e7f9513593cecf2ca95b9e2fa91e251b483
Implements: blueprint deprecate-rpc-backend
Dmitriy Rabotyagov