The hiera function is deprecated and does not work with the latest
hieradata version 5. It should be replaced by the new lookup
function[1].
[1] https://puppet.com/docs/puppet/7/hiera_automatic.html
With the lookup function, we can define value type and merge behavior,
but these are kept default at this moment to limit scope of this change
to just simple replacement. Adding value type might be useful to make
sure the value is in expected type (especially when a boolean value is
expected), but we will revisit that later.
example:
lookup(<NAME>, [<VALUE TYPE>], [<MERGE BEHAVIOR>], [<DEFAULT VALUE>])
This covers the remaining manifests to set up pacemaker resource.
Change-Id: I749b979a7333f68a646f36afa912603b1af0a943
... because Docker support has been removed from tht and these are no
longer used.
Depends-on: https://review.opendev.org/843755
Change-Id: I5719d06464ba2c1d37898b44f70ac5521ceaaf7e
Depending on the host history, it may happen some directory content
don't have the correct SELinux type. This has been seen with OVN
service, during a Queens -> Train FFU:
while the /var/lib/openvswitch/ovn directory had the correct
container_file_t type, some files in this location were typed with
openvswitch_var_lib_t, leading to errors during the deploy part of the
upgrade (after the OS upgrade, when the deploy is running on the cleaned
host).
The specific issue depends on the actual files with the wrong label, but
usually it involves a container crash/error, leading to a deploy error,
and a manual intervention in order to correct the SELinux type in the
location.
This situation may happen when first deployed on Queens, since it was
using Docker. For the records, back then Docker Daemon was configured in
order to disable the SELinux support, so it didn't really care about
labels; but the situation is different with Podman, and we have a full
SELinux support at all levels on the OS, leading to the issue.
For the records, tripleo-heat-templates as well as tripleo-ansible are
setting the "setype: container_file_t" on the directories, but we don't
use the "recurse: true" in order to avoid performance issues - some
locations might be huge, and it would take too much time to relabel
everything via ansible.
This patch aims to converge all the mounts to the same options, and
ensure no SELinux denial can prevent the actual container startup and
function.
Change-Id: Ic3e427156fc82c524c763d1896937fcc3c49fabb
Closes-Bug: #1943459
In tripleo-heat-templates a new 'CephConfigPath' parameter
is introduced with the purpose of customizing the path where the
Ceph config and keyring files are created on the host.
This change makes sure that puppet-tripleo is able to consume
a custom location for the Ceph config and keyring files.
Closes-Bug: #1708302
Co-Authored-By: Giulio Fidente <gfidente@redhat.com>
Change-Id: Iaabb66cd26f0246defe391a4e34f4eab3c3c5fee
When podman dropped the journald log-driver we rushed to move to the supported
k8s-file driver. This had the side effect of us losing the stdout logs of the
HA containers.
In fact previously we were easily able to troubleshoot haproxy startup failures
just by looking in the journal. These days instead if haproxy fails to start we
have no traces whatsoever in the logs, because when a container fails it gets
stopped by pacemaker (and consequently removed) and no logs on the system are
available any longer.
Tested as follows:
1) Redeploy a previously deployed overcloud that did not have the patch
and observe that we now log the startup of HA bundles in /var/log/containers/stdouts/*bundle.log
[root@controller-0 stdouts]# ls -l *bundle.log |grep -v -e init -e restart
-rw-------. 1 root root 16032 Apr 14 14:13 openstack-cinder-volume.log
-rw-------. 1 root root 19515 Apr 14 14:00 haproxy-bundle.log
-rw-------. 1 root root 10509 Apr 14 14:03 ovn-dbs-bundle.log
-rw-------. 1 root root 6451 Apr 14 14:00 redis-bundle.log
2) Deploy a composable HA overcloud from scratch with the patch above
and observe that we obtain the stdout on disk.
Note that most HA containers log to their usual on-host files just
fine, we are mainly missing haproxy logs and/or the kolla startup only
of the HA containers.
Closes-Bug: #1872734
Change-Id: I4270b398366e90206adffe32f812632b50df615b
Downcase in puppet 6.14 throws an error if the input to it is Undef. We
can avoid this by checking for a value before trying to downcase.
See context https://review.rdoproject.org/r/#/c/26297/
Change-Id: Ib2e97060523a4198a14949a15c9171b56928699c
Allow all bundles --user option to be overridden as some of them might
prefer switching to a non-root user when possible.
The ovn-dbs bundle is a bit special because it never specified any user.
Hence we default that user to undef and do not set anything.
Tested as follows:
1. deployed an overcloud
2. patched it with this change
3. redeployed and and then observed that no HA container has restarted at all
4. verified cinder-volume runs with root by default:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 4204 716 ? Ss 09:01 0:00 dumb-init --single-child -- /bin/bash /usr/local/bin/kolla_start
root 7 0.7 0.7 912976 145760 ? S 09:01 1:04 /usr/bin/python3 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf
root 71 0.1 0.6 925800 124640 ? S 09:01 0:14 /usr/bin/python3 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf
5. added 'tripleo::profile::pacemaker::cinder::volume_bundle::bundle_user: cinder' to
the templates and redeployed
6. Observed that cinder-volume got restarted and now runs with cinder
user:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
cinder 1 0.0 0.0 4204 804 ? Ss 12:23 0:00 dumb-init --single-child -- /bin/bash /usr/local/bin/kolla_start
cinder 7 2.1 0.7 912976 145432 ? S 12:23 0:04 /usr/bin/python3 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf
cinder 64 0.3 0.5 919908 118452 ? S 12:23 0:00 /usr/bin/python3 /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf
Change-Id: I985d0d192ef3accf7fdd31503348de80713fded4
Currently in puppet-tripleo for the HA container we hardcode the following:
options => "--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS${tls_priorities_real}",
Since at least podman had some changes in terms of supported driver
backends (and bugs) it's best if we make this configurable. While we're
at it we should also switch to k8s-file as a driver when podman is being
used which is what all other containers are using. When docker is the
default container_cli we will stick to journald as usual.
Tested this on a Train environment and successfully verified that
we still see the correct logs in /var/log/containers/.../...
Change-Id: I5b1483826f816d11a064a937d59f9a8f468315a5
Closes-Bug: #1853517
We are transitioning from an array to an hash for the container
environment of each container:
I894f339cdf03bc2a93c588f826f738b0b851a3ad
Mainly to make it consummable by Ansible later; where the
podman_container module needs a dict instead of a list.
This patch just changes the default, and also adds support for an Hash
instead of a List, but still supporting the List.
Change-Id: I4e53a4a3464940660473bcbe74e30507a69a4019
We add initial support for being able to specify tls priorities in
pacemaker. For bundles this will happen via an env variable because
pacemaker_remote is started normally as a process and there is no
sourcing of /etc/sysconfig/pacemaker.
Tested on both queens and stein. Via a deploy and a redeploy against
existing cloud. Observed that:
A) We got PCMK_tls_priorities inside /etc/sysconfig/pacemaker with the
value that was passed in THT
B) Containers had the following env variable set:
"PCMK_tls_priorities=normal",
The '-e' addition is a noop in case the PCMK_tls_priorities is unset
so that we do not change the signature of the resources and hence do
not needlessly restart the HA resource.
Depends-On: I1971810f6a90f244ed5ced972a5fe7fde29dde86
Change-Id: I703b5a429f48063474aace85bc45d948f5c91435
For the upgrade we have to re-provision the controller cluster, one
node at a time.
Using extra override variable set in hiera we are able to specify to
pacemaker which nodes should be added to the cluster.
Change-Id: I2f6ef4679265718fbbe8726ee6c81832bc468f3e
Implements: blueprint upgrades-with-os
We added a container backend in puppet-pacemaker via
Ia4a7b58d14d80e85d51e98acec1aad2ba90b69de. Let's now
let tripleo override it when needed.
Tested this via some hiera keys overrides and it works correctly.
Change-Id: I610923327462b901840131316a4984c8fe98faaa
When using the BlockStorage role there is one cinder volume per node and
those are not managed by pcmk. So when we force the property for all
nodes that have cinder_volume we will fail because pcmk is not running
on those nodes. Let's not set properties for nodes that are not running
pacemaker.
While we're at it let's remove *_nodes_count which are not used anyway.
Closes-Bug: #1786412
Change-Id: I42e9f3244bad60b5df2dfa940f132f72c606620e
When deploying a stack that containes mixed-case hostnames
the following error might be triggered:
Debug: try 15/20: /usr/sbin/pcs -f
/var/lib/pacemaker/cib/puppet-cib-backup20180405-8-1sqw3dc property set
--node TEST-STACK34-controller-1 redis-role=true
Debug: Error: Error: unable to set attribute redis-role
Could not map name=TEST-STACK34-controller-1 to a UUID
while the name in the cluster is test-stack34-controller-1
This used to work pre-bundles because we used the facter provided
$::hostname variable which was lower-cased for us. With bundles we
switched to setting cluster properties from the service bootstrap nodes
and so we used the '<service>_short_node_names' hiera key which might
contain mixed-case hostnames.
In order to fix this we just downcase() the short_node_names hiera
string that we loop on so we can get the same behaviour we had on bare
metal.
Tested on an env with mixed-case hostnames:
[root@uppercaseovercloud-controller-0 keystone]# hiera -c /etc/puppet/hiera.yaml rabbitmq_short_node_names
["UPPERCASEOverCloud-controller-0",
"UPPERCASEOverCloud-controller-1",
"UPPERCASEOverCloud-controller-2"]
Cluster pcs properties were set correctly:
[root@uppercaseovercloud-controller-0 keystone]# pcs property |grep rabbitmq
uppercaseovercloud-controller-0: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-0
uppercaseovercloud-controller-1: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-1
uppercaseovercloud-controller-2: galera-role=true haproxy-role=true rabbitmq-role=true redis-role=true rmq-node-attr-last-known-rabbitmq=rabbit@uppercaseovercloud-controller-2
Co-Authored-By: Damien Ciabrini <dciabrin@redhat.com>
Depends-On: Ie240b8a4217827dd8ade82479a828817d63143ba
Closes-bug: #1773219
Change-Id: I5bd49c4a1b13b2310f8a1173aa6b86abfa5dab3d
Add parameters for controlling the docker container settings used to
create the cinder-volume and cinder-backup pacemaker bundles. The
parameters eliminate the need to hard-code the list of docker volumes
and environment variables, making it possible to control the values
using hiera data.
For backward compatibility, the previous hard-coded values are used
when no parameter inputs are supplied.
Partial-Bug: #1748290
Change-Id: I4ba0d78ad17183b97290b853a6c103e55bc8977c
This adds the same CA cert mounts which other pacemaker managed
containers like rabbitmq, redis, and haproxy.
With this change, cinder-backup should work correctly when running SSL
enabled.
Change-Id: I199c03ba36a24e6b1caf535ed285047952ac9eb0
Closes-Bug: #1747326
Some pacemaker bundle resources were in conflict and that made Puppet
catalog failing. This patch makes sure that resources are now unique.
Change-Id: I940cec6d670df39ac6e2a3559a028acbeee99331
Closes-Bug: #1742795
Containerized or running on baremetal services
on a node must use the same iSCSI Qualified Name (IQN).
However, overcloud nodes must have a unique IQN.
Tht's puppet config bind mounts the real hosts' /etc/iscsid
directory so that puppet ensures the IQN is unique and
is reset once, and only once.
Switch the host path bind mount for cinder bundles to catch up
that configuration path as well.
Related-bug: #1735425
Change-Id: I7e9f0641164691682516ac3e72e2145c7d112409
Co-authored-by: Alan Bishop <abishop@redhat.com>
Co-authored-by: Martin André <m.andre@redhat.com>
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
This solves a problem with bind-mounts when the containers are holding
files descriptors open.
At the same time this makes the template more robust to puppet changes
since new config files will be available in the containers without
needing to update the templates.
Closes-Bug: #1698323
Change-Id: I857c94ba5f7f064d7c58df621ec5d477654b9166
Depends-On: I78dcec741a941dc21adba33ba33a6dc6ff1d217c
The step is typically set with the hieradata setting an integer value:
{"step": 1}
However it would be useful for the value to be a string so that
substitutions are possible, for example:
{"step": "%{::step}"}
This change ensures the step parameter defaults to an integer by
calling Integer(hiera('step'))
This change was made by manually removing the undef defaults from
fluentd.pp, uchiwa.pp, and sensu.pp then bulk updating with:
find ./ -type f -print0 |xargs -0 sed -i "s/= hiera('step')/= Integer(hiera('step'))/"
Change-Id: I8a47ca53a7dea8391103abcb8960a97036a6f5b3
This module is used by tripleo-heat-templates to configure and deploy
Kolla-based cinder-volume containers managed by pacemaker.
We use short-lived containers that call pcs via puppet to create
the needed pacemaker resources, properties and constraints.
Co-Authored-By: Michele Baldesari <michele@acksyn.org>
Partial-Bug: #1668920
Change-Id: I95ad4dd89b47396bea672813d87de35e64c04b2d
Ghanshyam Mann