This allows us to continue execution until a certain proportion of hosts
to fail. This can be useful at scale, where failures are common, and
restarting a deployment is time-consuming.
The default max failure percentage is 100, keeping the default
behaviour. A global max failure percentage may be set via
kolla_max_fail_percentage, and individual services may define a max
failure percentage via <service>_max_fail_percentage.
Note that all hosts in the inventory must be reachable for fact
gathering, even those not included in a --limit.
Closes-Bug: #1833737
Change-Id: I808474a75c0f0e8b539dc0421374b06cea44be4f
This commit adds SystemdWorker class to kolla_docker ansible module.
It is used to manage container state via systemd calls.
Change-Id: I20e65a6771ebeee462a3aaaabaa5f0596bdd0581
Signed-off-by: Ivan Halomi <i.halomi@partner.samsung.com>
Signed-off-by: Martin Hiner <m.hiner@partner.samsung.com>
Kolla Ansible now supports failing execution early if fact collection
fails on any of the hosts. This is to avoid late failures due to missing
facts (especially cross-host).
Change-Id: I7a74b937ded0b9da0621cf413f3a5d0d13a2cd68
Partial-Bug: #1833737
Ansible facts can have a large impact on the performance of the Ansible
control host. This patch introduces some control over which facts are
gathered (kolla_ansible_setup_gather_subset) and which facts are stored
(kolla_ansible_setup_filter). By default we do not change the default
values of these arguments to the setup module. The flexibility of these
arguments is limited, but they do provide enough for a large performance
improvement in a typical moderate to large OpenStack cloud.
In particular, the large complex dict fact for each interface has a
large effect, and on an OpenStack controller or hypervisor there may be
many virtual interfaces. We can use the kolla_ansible_setup_filter
variable to help:
kolla_ansible_setup_filter: 'ansible_[!qt]*'
This causes Ansible to collect but not store facts matching that
pattern, which includes the virtual interface facts. Currently we are
not referencing other facts matching the pattern within Kolla Ansible.
Note that including the 'ansible_' prefix causes meta facts module_setup
and gather_subset to be filtered, but this seems to be the only way to
get a good match on the interface facts. To work around this, we use
ansible_facts rather than module_setup to detect whether facts exist in
the cache.
The exact improvement will vary, but has been reported to be as large as
18x on systems with many virtual interfaces.
For reference, here are some other tunings tried:
* Increased the number of forks (great speedup depending of the size of
the deployment)
* Use `strategy = mitogen_linear` (cut processing time in half)
* Ansible caching (little speed up)
* SSH tunning (little speed up)
Co-Authored-By: Mark Goddard <mark@stackhpc.com>
Closes-Bug: #1921538
Change-Id: Iae8ca4aae945892f1dc65e1b10381d2e26e88805
By default, Ansible injects a variable for every fact, prefixed with
ansible_. This can result in a large number of variables for each host,
which at scale can incur a performance penalty. Ansible provides a
configuration option [0] that can be set to False to prevent this
injection of facts. In this case, facts should be referenced via
ansible_facts.<fact>.
This change updates all references to Ansible facts within Kolla Ansible
from using individual fact variables to using the items in the
ansible_facts dictionary. This allows users to disable fact variable
injection in their Ansible configuration, which may provide some
performance improvement.
This change disables fact variable injection in the ansible
configuration used in CI, to catch any attempts to use the injected
variables.
[0] https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars
Change-Id: I7e9d5c9b8b9164d4aee3abb4e37c8f28d98ff5d1
Partially-Implements: blueprint performance-improvements
This trivial patch is just turning off ansible
changed report for group_by tasks as it could
be confusing for user.
Change-Id: I7512af573782359a6f01290a55291ac7eb0de867
One way to improve the performance of Ansible is through fact caching.
Rather than gather facts in every play, we can configure Ansible to
cache them in a persistent store. An example Ansible configuration for
doing this is as follows:
[defaults]
gathering = smart
fact_caching = jsonfile
fact_caching_connection = ./facts
fact_caching_timeout = 86400
This does not affect Kolla Ansible however, since we use the setup
module which unconditionally gathers facts regardless of the state of
the cache. This gets worse with large inventories limited to a small
batch of hosts via --limit or serial, since the limited hosts must
gather facts for all others.
One way to detect whether facts exist for a host is via the
'module_setup' variable, which exists only when facts exist. This change
uses the 'module_setup' fact to determine whether facts need to be
gathered for hosts outside of the batch. For hosts in the batch, we
switch from using the setup module to gather_facts on the play, which
can use the 'smart' gathering logic.
Change-Id: I04841fb62b2e1d9e97ce4b75ce3a7349b9c74036
Partially-Implements: blueprint performance-improvements
Prior to this change, when the --limit argument is used, each host in the
limit gathers facts for every other host. This is clearly unnecessary, and
can result in up to (N-1)^2 fact gathers.
This change gathers facts for each host only once. Hosts that are not in
the limit are divided between those that are in the limit, and facts are
gathered via delegation.
This change also factors out the fact gathering logic into a separate
playbook that is imported where necessary.
Change-Id: I923df5af41a7f1b7b0142d0da185a9a0979be543
Mark Goddard