All references to Gentoo, SUSE, Debian stretch and Centos-7 are removed.
Conditional tasks, ternary operators and variables are simplified where possible
OS specific variables files are generalised where possible
Change-Id: I2defac928ff0081b262ba31bdb9981274f13b32b
Spacing changes were inadvertently introduced in
https://review.opendev.org/742121 which result in duplicate entries
in LXC config files.
This commit reverts to the original formatting with spaces either
side of the '=', although it makes no attempt to fix files which
may already include duplicate entries.
Change-Id: Ia95bbc959b54f494b5861afcb4e84d4c227e7b31
The removal of support for Centos-7 means that it is no longer necessary
to maintain compatibility with LXC2 configuration keys. This patch removes
the code which substitutes LXC3 keys for LXC2 keys.
Depends-On: https://review.opendev.org/742166
Depends-On: https://review.opendev.org/742103
Change-Id: I2911a20a3391e880df80f41eed5c9a8d5e36c2f4
Precess column was present only for CentOS 8 and is not applicable for
other distros. In the meanwhile Active column is present in all distros.
Change-Id: I13605f21497c7eb8e2dd569ab90e2466bce8ac3e
There is a race condition between starting an lxc container and executing
the first ansible task. Ansible makes heavy use of /tmp and the first
task executed after 'lxc-start' will collide with systemd-tmpfiles-setup
which by default removes all content from /tmp, including the working
files of any ansible task which happens to be running. This causes a fatal
error for ansible which cannot be recovered with retries.
This patch adds a raw command to check the state of the tmpfiles-setup
service and wait until it has completed, avoiding the race confdition.
Co-Authored-By: Dmitriy Rabotyagov <noonedeadpunk@ya.ru>
Change-Id: I8111ae7548cddd71b0f384157e28ced40392401b
btrfs-tools is deprecated and replaced by btrfs-progs on modern
debian derivative operating systems.
Change-Id: Iaf5fb24146b7203e879feccc7c96b0984b425d01
Instead of using the Ansible distribution to find out whether lxc-3 is
installed, use the `lxc_three_syntax` variable.
Change-Id: I30f13043d614d6460ef492cfd3d869841fe3ba44
Signed-off-by: Nicolas Bock <nicolas.bock@suse.com>
Functional tests have been broken by [1] as it has prescedence over
host_vars which are defined for functional tests.
[1] https://review.opendev.org/#/c/707943/
Change-Id: If687504ed08c34bd3336d31b1ae044b7d633318c
The connection plugin no longer falls back on using the
inventory_hostname as the container_name. Set container_name as a host
var for each container in the test inventory.
Change-Id: Ic23a77d8f88ed890a219c410d37a94bd97037c53
This change allows the role to accept either lxc2 or lxc3 config
keys, plus a list of key substitutions to make when writing the
lxc config file.
This allows a set of config keys to remain defined as variables
outside this role and generate a valid config on both lxc2 and lxc3
based hosts.
Change-Id: Ifc871a9fcaf77ff36cfcc1c87b1f406862d46d22
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: Ie89ff6580bec52b598776c479a909c9a99c005b0
Now that bionic testing is added into the tests repos, we can
start testing it in the repo.
Since bionic uses lxc >= 3, we need to make some adjustments to the role
to allow the role to work with both lxc > 3 and lxc < 3, there were
several config options changes which will impact on upgradeability.
LXC >= 3 requires networks to have an index, we can achieve this by
taking the network dict and converting it to a list, and using those to
generate the id "with_indexed_items".
Depends-On: https://review.openstack.org/#/c/566959/
Depends-On: https://review.openstack.org/#/c/567038/
Change-Id: Ib80c2ed2a01a4a6a8c48aed9bdf9a50e45ea9564
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Now that run_tests.sh handles the tests repo clone, we can
remove the use of the older tests-repo-clone.sh script.
Change-Id: Iead678057f3888fe7aaddce6685865f4fcdfed53
The container and host can link journals giving operators the ability to
log stream and check on the health of a system without needing to login
(attach) to the container. This change implements journal linking for
LXC containers following the reference systemd specification.
Reference implementation:
https://www.freedesktop.org/software/systemd/man/systemd-nspawn.html#--link-journal=
Change-Id: Id68cf39a77b5dd9c13c010829b47cd7a414378bc
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Patch I0d83fd4895d4c5beaf5a84a239c1a1ed71521dee dropped the ARP=yes
option for networkd because it's not supported by old systemd releases.
This however brings back a problem where the default one sysctl
arp_notify option in the kernel may not correctly set for our use case.
Containers are created with random MAC addresses so we need to ensure
that ARP entries are populated correctly when a container is restarted.
Instead of having to implement some sort of a new workaround on the host,
it's probably better to create all containers with fixed MAC addresses from
now on.
Change-Id: I8ad390fc3ce27756f26c57c92aaa3adc8e506a17
We should use domain names for the external network testing task in
order to verify no only that the default gateway works properly but also
that our DNS is able to resolve hostnames.
Change-Id: I3aebcf1dff8268e4dbaebae8fb598ee27e3f481d
Depends-On: I316c3851f40f08d272b7bb5f7165e010e3a95c3a
Depends-On: Ied7632037f737c3f32c34dac70531065c54496c9
Depends-On: I14f8373897da28dea2ea03500c2be46c5b40d51c
Depends-On: I0d83fd4895d4c5beaf5a84a239c1a1ed71521dee
The ansible_distribution variable is causing some troubles since it can
contain spaces etc. As such, we can simply use the ansible_pkg_mgr
module to figure out the name of the package we want to install.
Change-Id: Ic92eb1f9030df2883b049b9868e031ff4f0d42f2
Unify container network interfaces using Systemd Networkd for ubuntu,
centos, and openSUSE. This change allows the role to use a single way to
configure container networks.
Care has been taken to ensure we're able to cleanly upgrade to the new
capabilities within existing environments without breaking any feature
compatibility or causing any container restarts.
It's also worth noting that all of the pre/post networking up/down
script options have been converted to systemd "oneshot" services. This
retains the ability to run adhoc scripts post network availability
while also opening up this capability, which used to be ubuntu only,
to all of our supported operating systems.
> Our usage of `lxc-attach` was removed in favor of `nsenter` to fix a
issue where multiple `lxc-attach` commands issued to a single physical
host could result in a hang.
> Scripts that were being generated inline have been placed into
template files. This solves a long standing memory consumption issue
when creating lots of containers. The old shell tasks will now be
executed from a generated script. While this should also help with
debugging, the main driver is to ensure better system stability.
> A lot of cleanup has been done throughout the task files and
templates. In the process of updating the role to use unified
networking a lot of duplicate tasks, scripts, and processes have
consolidated.
> Handlers have been added for network connection wait conditions and
to various service restarts.
> The OSA plugins have been added to this role as a dependency. We
rely on the connection plugins throughout the stack however we were
doing a lot of workarounds to cater to the possibility of a deployer
running this role without them. This change simply adds the plugins
as a known dependency which allows for a more streamlined setup.
Change-Id: I5d3ddcfa11d575648a69a04f2fb30236c2c89da3
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
The overlayfs backing store doesn't play well with the unconfined
profile and many tools (eg ping, traceroute) are failing to work
with the following error:
ping: error while loading shared libraries: libcap.so.2: cannot stat
shared object: Permission denied
As such, lets switch to the lxc-openstack profile is overlayfs is used
as the backing store.
Change-Id: Ibe1149ee4fedd2b3d487887e504c500c96165467
Related-Bug: #1612412
The handler would try and stop a container before restarting it however
if the container was already stopped the handler would fail instead of
simply moving on to the next task. This change makes the "stop" portion
of the task detect the return status code of "2" when restarting the
container. If the return code is "2" we know that the container is
already stopped and that no change has occurred.
For the sake of consistency and to ensure the greatest chance for
success the test task that stops a container has also been given the
same setup.
Change-Id: Ia4856f36b2d106d987e3c774f31493e25a23d4b5
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
With the merge of https://review.openstack.org/520177 in the
tests repo some ansible-lint failures which previously were
not being picked up are now detected.
This adds the appropriate skip tags to the tasks so that they
are not evaluated by ansible-lint.
Change-Id: Ia91f73d4f17e94a150c93c75c618778c25823d0d
This patch fixes an issue with the test package name.
Depends-On: I913284b0dc0165e102d4016760947223fb129a92
Change-Id: I84752bd83d76b7d5e7ac38a5c2b8a81d75d5ceb7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This patch implements an initial set of jobs intended to match
the current job execution method. It does not intend to improve
how the jobs are executed - only to replicate what is currently
in openstack-infra/openstack-zuul-jobs and provide the platform
to iterate on.
Change-Id: If86f31a6ff188c57c5981dcf9eddc26af7101b25
This change adds several tox functional sections to support our
different backend storage options. This change will allow us to create
jenkens jobs specific to the various backends to validate they work as
we expect them to.
Change-Id: Ic1152ae666973af8c01499bb5ca89d8eeaa5f1d2
Depends-On: I70d53cabd0888954f31def924e9f4436398cdebf
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
",create=dir" was added to the integrated repo in
I29be91b4dca5abe1bb8307429a86c06dbd3ef7aa to ensure that LXC handles
the bind mount directory creation inside the container.
Also a bug is fixed where bind mounts defined as part of the container
configuration list replace all '=' within the string as ' = '. To fix
this only the first = is padded with whitespace.
Change-Id: Ieed33a413ed9abec98e46cd519c18a95d5b6bd75
Some sysctl can be applied to containers, so we add a test
to prove our containers can do it.
Change-Id: I40e2f0af00d6d763efcbb07306791d3cd3feff0d
Fixes-Bug: #1685677
lxc_container_bind_mounts is overridden within the tests repo with
change I774343234a25063eb320cac85ba696d908f0a416.
Adjust the functional test to check for the overridden value.
Change-Id: Icaa1af40b159aaaa2b096c3148356f2d2abb38f5
Add support for SUSE based distributions. We also update the bindep.txt,
run_tests.sh, tests-repo-clone.sh and Vagrantfile files from the
openstack-ansible-tests repository.
Change-Id: I9ac018ac1a94dac74a2ef213dccedf95b4272134
We use ping to determine if the container has external connectivity.
We also use the management IP in Ansible to connect to the container
which is in a different interface than the external one. The management
interface has a static IP so it's available much earlier than the
external one which obtains the IP via DHCP. As such, in some cases,
the external interface may not be configured early enough and the ping
fails. We can improve this situation by sending more ICMP packets less
frequently to give the interface sometime to be configured properly.
This patch effectively allows 30 seconds for interface to reach a working
state to determine if there is external connectivity or not which should
be more than enough.
This fixes some random CI failures where one container has external
connectivity and another one doesn't even though they are configured
exactly the same.
For example:
2017-05-04 19:30:44.264567 | changed: [container1] => {
2017-05-04 19:30:44.264610 | "changed": true,
2017-05-04 19:30:44.264627 | "cmd": "ping -c 3 8.8.8.8",
...
2017-05-04 19:30:44.264900 | "stdout": "PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.\n\n--- 8.8.8.8 ping statistics ---\n3 packets transmitted, 0 received, 100% packet loss, time 1998ms",
2017-05-04 19:30:44.264911 | "stdout_lines": [
2017-05-04 19:30:44.264929 | "PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.",
2017-05-04 19:30:44.264937 | "",
2017-05-04 19:30:44.264952 | "--- 8.8.8.8 ping statistics ---",
2017-05-04 19:30:44.264974 | "3 packets transmitted, 0 received, 100% packet loss, time 1998ms"
2017-05-04 19:30:44.264981 | ]
but
2017-05-04 19:30:34.299060 | changed: [container2] => {
2017-05-04 19:30:34.299112 | "changed": true,
2017-05-04 19:30:34.299128 | "cmd": "ping -c 3 8.8.8.8",
...
2017-05-04 19:30:34.299683 | "stdout": "PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.\n64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=3.67 ms\n64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=3.65 ms\n64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=3.66 ms\n\n--- 8.8.8.8 ping statistics ---\n3 packets transmitted, 3 received, 0% packet loss, time 2003ms\nrtt min/avg/max/mdev = 3.656/3.664/3.678/0.070 ms",
2017-05-04 19:30:34.299697 | "stdout_lines": [
2017-05-04 19:30:34.299715 | "PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.",
2017-05-04 19:30:34.299735 | "64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=3.67 ms",
2017-05-04 19:30:34.299755 | "64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=3.65 ms",
2017-05-04 19:30:34.299775 | "64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=3.66 ms",
2017-05-04 19:30:34.299783 | "",
2017-05-04 19:30:34.299798 | "--- 8.8.8.8 ping statistics ---",
2017-05-04 19:30:34.299820 | "3 packets transmitted, 3 received, 0% packet loss, time 2003ms",
2017-05-04 19:30:34.299839 | "rtt min/avg/max/mdev = 3.656/3.664/3.678/0.070 ms"
2017-05-04 19:30:34.299845 | ]
Change-Id: I77eb168d6c3abc5e679c7dd60c52353dc44f8afb
When executing the tests repo clone in OpenStack-CI,
use zuul-cloner instead of git to enable cross-repo
testing. This ensures that if a dependent patch from
the tests repo is noted using 'Depends-On: <change-id>'
in the commit message, that patch will be included.
Change-Id: I89d06e204c64eda194da827c59e7506cac9b57f8
Depends-On: Idce7abebf32f24c356a27e099fbca954d917402b
Move test host vars from the inventory to individual files for each
host. 'ansible_become' has been removed from localhost's vars since it
should be handled on a playbook basis. The containers' management
addresses have also been moved to the br-mgmt bridge.
Change-Id: I521ca58d07a27278864b01e02c00db43b31a0916
Currently the entire environment file is replaced,
which may remove any existing configuration on the
hosts including any pre-existing proxy configurations
and anything else the deployer may have put there.
This method replaces it with an additive process which
respects pre-existing content, but still allows the
global_environment_variables variable changes to be
fully reflected in the resulting environment file.
As the PATH setting in the previous template was
the default path which will already be present in
the container, that has been removed from the template.
Change-Id: I930f1711fbd56d2c97e8c80bd990350fa0c7ba73
This change resolves a long standing issue where a container's mac
address regenerates when it was restarted. In most cases when a
container is restarted and it's mac address is rotated and nothing bad
happens; mac learning will resolve itself given enough time in just
about all situations. However services like neutron-agents are long
lived and are highly sensitive to network changes. These types of
services expect consistent hardware addressing and when mac
addresses rotate may become confused.
To limit the possibility of prolonged downtime caused by mac address
rotation on network sensitive containers an option has been created to
allow a container to have a fixed mac address. If this option is enabled
the container will generate fixed addresses for all networks assosiated
with the specific container. The option is `lxc_container_fixed_mac` and
it has a default value of "false".
Change-Id: Ie1a8dc172c45fc2b4cfa724a2bafa67cb481ba73
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This change removes the use of 'ignore_errors: true' because it causes deployers
to see red output and a stacktrace, which traditionally means something is broken,
even when the failure is known to have a fall back option or be intentional. This
conversion will provide a generally cleaner interface.
It should be noted that the 'failed' filter will still function normally. Tasks
with the 'failed_when: false' option will still be marked as 'failed' in any
registered variable. This change simply makes the output look cleaner.
Change-Id: Ia5c6d38a33e96c969116606859a9a60c6a79a663
Closes-Bug: #1633438
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
This patch consumes the test scripts implemented by
https://review.openstack.org/375061 to ensure that
the tests and test preparation is consistent and
more maintainable.
Change-Id: I8265d00c0e7c6ba9ee483098effed4e20027398b
Ansible 2.1.1 introduces a regression in the way conditional
includes are handled which results in every task in the
included file being evaluated even if the condition for the
include is not met. This extends the run time significantly
for a deployment.
This patch forces all conditional includes to be dynamic.
Change-Id: Ic9d45aea1b930ef7ffee00628cf2993ad1b79e14
Related-Bug: https://github.com/ansible/ansible/issues/17687
Dmitriy Rabotyagov