Commit Graph

136 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov 0ba35bf841 Add quorum queues support for service
This change implements and enables by default quorum support
for rabbitmq as well as providing default variables to globally tune
it's behaviour.

In order to ensure upgrade path and ability to switch back to HA queues
we change vhost names with removing leading `/`, as enabling quorum
requires to remove exchange which is tricky thing to do with running
services.

Change-Id: Id5f6cabed7ec035845865d6d5facc63590c56d43
2023-11-13 12:11:44 +00:00
Dmitriy Rabotyagov 99c6fb2835 Fix linters and metadata
With update of ansible-lint to version >=6.0.0 a lot of new
linters were added, that enabled by default. In order to comply
with linter rules we're applying changes to the role.

With that we also update metdata to reflect current state.

Depends-On: https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/888223
Change-Id: I6bbb97cd2f1daac42269a335014eb6cf7f28d24e
2023-07-17 11:45:20 +02:00
Damian Dabrowski 96a262b26b Add TLS support to swift backends
By overriding the variable `swift_backend_ssl: True` HTTPS will
be enabled, disabling HTTP support on the swift backend api.

The ansible-role-pki is used to generate the required TLS
certificates if this functionality is enabled.

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/879085
Change-Id: Idb7882775a90ada9bb9e1450168916c73bf8ae4b
2023-04-29 18:43:51 +02:00
Dmitriy Rabotyagov 5139ecc233 Support service tokens
Implement support for service_tokens. For that we convert
role_name to be a list along with renaming corresponding variable.

Additionally service_type is defined now for keystone_authtoken which
enables to validate tokens with restricted access rules

Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/845690
Change-Id: If34e0170ea0e0f7727cfadba982f3c7dae6ae216
2022-06-15 19:52:02 +02:00
Dmitriy Rabotyagov 451678a6fd Refactor definition of lock path
Depends-On: https://review.opendev.org/c/openstack/openstack-ansible/+/819300
Depends-On: https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/819298
Change-Id: I6ce09e0e0b83e14fc9386fc4cb49921a21fcabd4
2021-11-30 12:42:47 +02:00
Zuul 2ec9981fb2 Merge "Add variables for rabbitmq ssl configuration" 2021-05-18 14:15:13 +00:00
Jonathan Rosser acdaeda64b Add variables for rabbitmq ssl configuration
Change-Id: I0a1104f8312b16f618bdd8bf6b440730213c5fad
2021-05-17 12:10:40 +00:00
Dmitriy Rabotyagov 2ce00d6c69 Do not collect gnocchi project ID
Ceilometer middleware now accepts also project names,
so it's not required to gain project UUID [1].

We also switching gnocchi project name to `service` for all scenarios
as it's already whitelisted by default [2]

[1] e2bf485044
[2] 082dabb1b7/ceilometermiddleware/swift.py (L158)

Closes-Bug: #1879192
Change-Id: I40d3178d1b3bd1234ac34c6fa1d0d2bcb7160720
2021-05-13 14:39:43 +00:00
Dmitriy Rabotyagov 24bbb72ac9 Revert "split templates to work around configparser bug"
This reverts commit 3e151d97ad.

Reason for revert: Upstream bug has been fixed

Change-Id: I5498030487677523ae71ae2f9a13074ab4552204
Related-Bug: #1872553
Closes-Bug: #1921354
2021-03-25 10:16:08 +00:00
Zuul 992ab38b61 Merge "Use ansible_facts[] instead of fact variables" 2021-03-16 16:04:49 +00:00
Jonathan Rosser 440a53a4e6 Use ansible_facts[] instead of fact variables
See https://github.com/ansible/ansible/issues/73654

Change-Id: Ia3c80d5c0e63ee9c38868c41c9235c51a2498971
2021-03-16 08:20:27 +00:00
Jonathan Rosser 3459f10c27 Switch default virtualenv to python3
Change-Id: I5bc095b4ebcfa7e69ce34e058a438b10ec60a07a
2021-03-10 09:04:10 +00:00
Jonathan Rosser baa3dc3578 Move swift packages from constraints to requirements
This is necessary to use the new pip resolver

Change-Id: I5c22a77fc889b78bad6d1e27f7c631ef4b8a8220
2021-01-25 10:31:19 +00:00
Dmitriy Rabotyagov d3f80f734a Use global service variables
Instead of overriding each service separatelly it might make
sense for deployers to define some higher level variable that
will be used first or fallback to default variable.

Change-Id: Ia08e7c0c1a7398c6edac32dc04cb90e50b502059
2021-01-08 18:55:00 +02:00
Dmitriy Rabotyagov 5e4c9582f1 Reduce number of processes on small systems
Even the most modest 4C/8T system would run with the maximum 16 processes
due to the calculation being VCPU*2.

We devide amount of CPUs to number of threads for hyperthreaded CPUs

Change-Id: Ie3c590d413b001ac9ccdb6522c9654b4372b5e10
2020-11-30 14:01:40 +02:00
Dmitriy Rabotyagov 1d57000f24 Add global override for service bind address
Change-Id: Id04786c64e6dcc955e086a26f0abe38e2437d23f
2020-10-21 10:30:11 +03:00
Zuul 01bc0f8ff6 Merge "split templates to work around configparser bug" 2020-05-15 07:28:34 +00:00
Dmitriy Rabotyagov 2c2d130efe Cleanup after repo_build and pip_install retirement
Change-Id: Ic09462a1c0a6596c0846ff6241664f706718ff66
2020-05-12 23:06:31 +03:00
Matthew Thode 3e151d97ad
split templates to work around configparser bug
Move the percent character to the template from the string to allow
setting either % or %% as needed to work around the bug linked below.
This requires each service 'family' to have two versions of it's config
file.

Bug: https://bugs.launchpad.net/swift/+bug/1872553

Change-Id: I68f276224c51d5682d77123eae697767fadf9b19
Signed-off-by: Matthew Thode <mthode@mthode.org>
2020-04-16 13:49:01 -05:00
Dmitriy Rabotyagov be9ad06226 Fix replicator services
Current services were infinitelly spawning replicator process
instead running a server with another config file

Change-Id: I3310c11c0be38ae72b1f0bf94a849587fb8e5cf9
Closes-Bug: #1859159
2020-04-15 07:00:12 +00:00
Guilherme Steinmüller 677ab44170 Refactor memcached_servers
This patch aims to add a prefix for memcached_server
on each role to give the ability for deployers to
override the location of memcached cluster. I.e users
wants to create a single memcached cluster with k8s
for each service.

We also add pymemcache based on [1]

[1] https://review.opendev.org/711429

Change-Id: If17dd627708f03824939ba062b498675253b11d7
2020-03-16 15:05:19 +00:00
Dmitriy Rabotyagov 0832b3a665 Replace git.openstack.org with opendev.org
This patch replaces git.openstack.org with opendev.org as redirection
from old path was enabled.
Also we change upper constraints url due to [1]

[1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html

Change-Id: I3c897a4aac58aeb8ea641839e554c49d5eacb708
2019-12-04 18:22:58 +00:00
Dmitriy Rabotyagov (noonedeadpunk) ff6aa7a7fe Revert "Update fallocate_reserve to work with py3"
We need to revert this commit once related bug will be resolved

This reverts commit c1bdbfbb14.

Change-Id: Ic369410130fe94d18a5198b9d50b584c1935c1b6
Related-Bug: https://bugs.launchpad.net/swift/+bug/1844368
2019-09-19 10:13:37 +00:00
Dmitriy Rabotyagov c1bdbfbb14 Update fallocate_reserve to work with py3
As ConfigParser tries to convert "%" as a part of the variable[1],
we need to adjust it's default variable inside template for swift
to work with py3 unless this will be fixed in swift by using raw [2]

[1] https://docs.python.org/3.6/library/configparser.html#configparser.ConfigParser.get
[2] https://github.com/openstack/swift/blob/master/swift/account/server.py#L66

Change-Id: I9bbd3195a7b02a55e38207aa4bba182a6e58346c
Related-Bug: https://bugs.launchpad.net/swift/+bug/1844368
2019-09-17 19:51:59 +03:00
Jonathan Rosser 119bc32454 Allow venv python interpreter to be overridden
Change-Id: I7d5cfc8f07b92a71150a189eb352b8a02051c0c9
2019-09-09 09:31:44 +01:00
Dmitriy Rabotyagov d0fac1b559 Use systemd-journald instead of log files
This patch aims to migrate service from usage of regular syslog files
to journald.
By this we mean dropping rsyslog client installation. log_address is set
by default to /dev/log, which is served by journald.

Change-Id: I6dd0d77004394bb1ad674b53538b0679b056bb0f
2019-07-19 15:10:49 +03:00
Mohammed Naser 4de1219730 swift3: remove support
The swift3 middleware has been deprecated in replacement with the
s3api middleware instead.  This removes all the swift3 references
to hopefully enable someone to cleanly add s3api eventually.

Change-Id: I3a8a1ab861ec81b1f4f8dbc02a6a332d4ce495b8
2019-03-29 09:41:50 -04:00
Mohammed Naser 52aacc5f0b Update role for new source build process
The variables swift_developer_mode and swift_venv_download
no longer carry any meaning. This review changes swift to
do the equivalent of what developer_mode was all the time,
meaning that it always builds the venv and never requires
the repo server, but it will use a repo server when available.

As part of this, we move the source build out of its own file
because it's now a single task to include the venv build role.
This is just to make it easier to follow the code.

Change-Id: I9233e4daff0034339750477fd21d5cfa181afd83
2019-03-29 03:10:31 +00:00
Jesse Pretorius c7e11dfcbd Enable overriding the service setup host python interpreter
In order to enable the service setup host python interpreter to
be changed easily, we make it a variable. This will be useful
when someone sets the service setup host to be the utility
container, because we'll be able to set this var by default.

Change-Id: Icb3ea9ad782218c357ec0e3c577ecbe6fbf60461
2018-11-30 16:42:46 +00:00
Jonathan Rosser 87fb727472 Correct swift replication service units
These pointed to the wrong binaries, and in the case of the object
replicator, a non-existant binary. The names of the binaries have
been checked here:

https://packages.ubuntu.com/bionic/all/swift-account/filelist
https://packages.ubuntu.com/bionic/all/swift-container/filelist
https://packages.ubuntu.com/bionic/all/swift-object/filelist

Change-Id: Ib8d4875d44130048c5a5d8b2973d6da3a4321abe
2018-11-02 20:47:15 +00:00
Jesse Pretorius 7126647d7d Use a common python build/install role
In order to radically simplify how we prepare the service
venvs, we use a common role to do the wheel builds and the
venv preparation. This makes the process far simpler to
understand, because the role does its own building and
installing. It also reduces the code maintenance burden,
because instead of duplicating the build processes in the
repo_build role and the service role - we only have it all
done in a single place.

We also change the role venv tag var to use the integrated
build's common venv tag so that we can remove the role's
venv tag in group_vars in the integrated build. This reduces
memory consumption and also reduces the duplication.

This is by no means the final stop in the simplification
process, but it is a step forward. The will be work to follow
which:

1. Replaces 'developer mode' with an equivalent mechanism
   that uses the common role and is simpler to understand.
   We will also simplify the provisioning of pip install
   arguments when doing this.
2. Simplifies the installation of optional pip packages.
   Right now it's more complicated than it needs to be due
   to us needing to keep the py_pkgs plugin working in the
   integrated build.
3. Deduplicates the distro package installs. Right now the
   role installs the distro packages twice - just before
   building the venv, and during the python_venv_build role
   execution.

Depends-On: https://review.openstack.org/598957
Change-Id: Iecb64d28afe3acfbae7060af55c1a891310e5ef4
Implements: blueprint python-build-install-simplification
Signed-off-by: Jesse Pretorius <jesse.pretorius@rackspace.co.uk>
2018-09-03 11:07:02 +00:00
Zuul c3c1bffde9 Merge "Revert "Unblock gate: Provide default rabbitmq password"" 2018-08-07 19:46:35 +00:00
Jesse Pretorius (odyssey4me) 941febe046 Revert "Unblock gate: Provide default rabbitmq password"
This reverts commit 6ee96f224c.

This variable is no longer referenced anywhere, so we can remove it.

Change-Id: I70a262020918d51b20486cf819dc9131abfce7f1
2018-08-07 12:31:20 +00:00
Zuul 9df587fa4d Merge "Execute service setup against a delegated host using Ansible built-in modules" 2018-08-01 15:00:03 +00:00
Jesse Pretorius fc6c1935d7 Execute service setup against a delegated host using Ansible built-in modules
In order to reduce the packages required to pip install on to the hosts,
we allow the service setup to be delegated to a specific host, defaulting
to the deploy host. We also switch as many tasks as possible to using the
built-in Ansible modules which make use of the shade library.

The 'virtualenv' package is now installed appropriately by the openstack_hosts
role, so there's no need to install it any more. The 'httplib2' package is a
legacy Ansible requirement for the get_url/get_uri module which is no longer
needed. The keystone client library is not required any more now that we're
using the upstream modules. As there are no required packages left, the task
to install them is also removed.

Depends-On: https://review.openstack.org/582359
Depends-On: https://review.openstack.org/587376
Change-Id: I68f3a0bf2b7a3a12cbf40d7d6a853d5b4c6dd0f3
2018-07-31 17:37:02 +00:00
Jesse Pretorius 6ee96f224c Unblock gate: Provide default rabbitmq password
The integrated build has no value for swift_rabbitmq_password
and yet the oslomsg vars reference it. Given that the integrated
gate is quite hard to merge anything into right now - set a
default value here temporarily until we can merge a fix there.

Change-Id: I8c13fe063cd5ec454196bc630cb6362f97c4f146
2018-07-31 13:23:55 +01:00
Zuul d97fa0fd53 Merge "Move MQ vhost/user creation into role" 2018-07-28 08:20:19 +00:00
Jesse Pretorius 1ddd7590cf Move MQ vhost/user creation into role
There is no record for why we implement the MQ vhost/user creation
outside of the role in the playbook, when we could do it inside the
role.

Implementing it inside the role allows us to reduce the quantity of
group_vars duplicated from the role, and allows us to better document
the required variables in the role. The delegation can still be done
as it is done in the playbook too.

In this patch we implement the new variable:
- swift_oslomsg_notify_setup_host

This is used in the role to allow delegation of the MQ vhost/user
setup for each type to any host, but they default to using the first
member of the applicable oslomsg host group.

We also adjust some of the defaults to automatically inherit existing
vars set in group_vars form the integrated build so that we do not
need to do the wiring in the integrated build's group vars. We still
default them in the role too for independent role usage.

Depends-On: https://review.openstack.org/584630
Change-Id: I3d5e00e090d37ea7aa95460965749ef066b63b23
2018-07-26 09:43:10 +00:00
Jean-Philippe Evrard 9379dc8aee Remove httplib
The 'httplib2' package is a legacy Ansible requirement for the
get_url/get_uri module which is no longer needed.

Change-Id: I420dfd74e78963486ed9d5a66b8c3b72372b5ae6
2018-07-25 07:39:18 +00:00
Zuul cf7d3077a9 Merge "Update to use oslo.messaging service for Notify" 2018-07-02 10:19:57 +00:00
Kevin Carter 6175ff1e8d
Add packages required for osprofiler
The following packages are required in-order to run osprofiler.
these packages will provide deployers the ability to profile
a service on demand should they choose to enable the profile
functionality.

Change-Id: If60f98fc069c040680f58658aacc63a156c4317f
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-06-11 22:59:09 -05:00
Andrew Smith 91af246bdf Update to use oslo.messaging service for Notify
This introduces oslo.messaging variables that define the Notify transport
in place of the rabbitmq values.

This patch:
* Add oslo.messaging variable for Notify to defaults
* Update url generation
* Add oslo.messaging to inventory
* Add release note

Change-Id: I1c2e844c4c7a2256087bcc4521f970ca8e8c6b16
2018-06-11 16:45:16 -04:00
Markos Chandras 4bf5de8d30 Do not pip install keystoneclient on the host
The keystoneclient package is being installed on the host by PIP but
that means that a whole bunch of required dependencies are being pulled
in as well.

This brings the host to a rather messed up state when installing
keystone from distro packages, since distribution and
PIP packages are being mixed together. We only need the client to
register the service with keystone so we can simply use the distro
package for that to avoid installing lots of PIP packages on the
host.

Change-Id: Id5d79db00e1a4aa4983aafd92c088ef8f13a7da0
Implements: blueprint openstack-distribution-packages
2018-06-04 08:41:20 +01:00
Markos Chandras 51ea328b7c Add support for using distribution packages for OpenStack services
Distributions provide packages for the OpenStack services so we add
support for using these instead of the pip ones. However, functional
testing is not complete yet since it requires tempest to be in the swift
virtual environment which doesn't exist for distro installs. As such,
for functional testing to cover this method of installation it needs
to be changed quite a bit.

Change-Id: Id75e56440685df407a9991018dc07b5a75429082
Implements: blueprint openstack-distribution-packages
2018-06-04 08:41:19 +01:00
Markos Chandras ae8e01aada defaults: Do not install the cinderclient package
Nothing in the swift role needs the cinderclient package so there is
no need to install it.

Change-Id: I01b2a71fa185745b1b73546222779ef71889258a
2018-05-31 09:26:57 +01:00
Kevin Carter 8f1cb4dde5 Convert role to use a common systemd service role
This removes the systemd service templates and tasks from this role and
leverages a common systemd service role instead. This change removes a
lot of code duplication across all roles all without sacrificing features
or functionality. The intention of this change is to ensure uniformity and
reduce the maintenance burden on the community when sweeping changes are
needed. The exterior role is built to be OSA compatible and may be pulled
into tree should we deem it necessary.

Change-Id: Icb7ca523cb19c560de5c84b0d60a06305029192c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-04-22 14:00:27 +00:00
Zuul 0f26b461f8 Merge "Replace virtualenv-tools by a script" 2018-01-16 13:24:18 +00:00
Jean-Philippe Evrard d5ad55cf4c Replace virtualenv-tools by a script
virtualenv-tools has a bug which gets triggered in gates: it can't
change the shebang of a virtualenv python bin/ files if they
were generated with a virtualenv script whose shebang ends with
python2 instead of python.

Because we can't modify virtualenv-tools, we use shell scripts
instead.

Change-Id: I2dc9a507162fcd2323e19a3a2daa97c6db9cdcdb
Partial-Bug: #1741634
2018-01-15 14:17:07 +00:00
Jean-Philippe Evrard 33505b9482 Update PyPy version
This is the time of the cycle where we update all the static
elements in the roles. PyPy has a static file version, so we
bump it.

Change-Id: I15c21c177c32dcae0a7f2710b50f91829be09586
2017-11-27 09:41:08 +00:00
Charles Farquhar 8b2fc7afc7 Fix ordering of swift3 in middleware pipeline
Beginning with commit 6ffcc29 of swift3, the swift3 and s3token middlewares
must come between authtoken and keystoneauth in the swift proxy pipeline.

When 6ffcc29 was committed, reordering was optional at the expense of an
unneceessary call to keystone.  However, the change becomes mandatory when using
keystone v3 tokens.  Without this change, authtoken will strip the necessary
headers that s3token just added to the environment.

Change-Id: Ic7d07d869aa617ee00190a9aedf411017539b97b
2017-07-27 15:52:58 -05:00