Commit Graph

30 Commits

Author SHA1 Message Date
Dmitriy Rabotyagov 4ce9e713ce Make plugins usable as part of collection
Since we're converting plugins repo to collection, we make plugins
usable from collection. For that we change repo structure.

Symlinks to old destinations are created for backwards compatability

Change-Id: I15b4a9f50910794c11b57457f9f4fbb34f321654
2022-01-12 13:41:09 +00:00
Jonathan Rosser d16dbe169c Remove chroot connection from ssh plugin
This is not used anywhere in OSA so remove to simplify the code

Change-Id: I4a4fcdadb4408960aa885269d573e66dd8542f0e
2021-01-14 08:38:56 +00:00
Dmitriy Rabotyagov a695e3429c Replace deprecated imp module with importlib
The imp module is deprecated since version 3.4, so we should
use importlib wich is recommended replacement.

Change-Id: Ic725bf6e99af11bee6a4c4a5c8f409f0675ce8bf
2020-08-31 09:35:03 +00:00
Zuul f3ae47e349 Merge "Do not use paramkio transport for delegated tasks" 2020-07-06 15:50:58 +00:00
Jonathan Rosser 871d91cf26 Optimise linear strategy access to physical host variables
Change from using variable manager to the var we already have for
the physical host. This saves nearly 50% time for re-running
playbooks/lxc-container-create.yml

Change-Id: Ic50763f9c74746ef94925a4dd52d16e1f313711b
2020-06-24 20:13:53 +03:00
Jonathan Rosser 15ee001183 Do not use paramkio transport for delegated tasks
This saves 25s from the previous runtime of 4m21s for
playbooks/lxc-containers-create on an already deployed AIO.

Change-Id: I8b315efed26a22ae669432856eb1e7d994f4ab3
2020-06-22 14:53:40 +01:00
wangqi 12d5b33dc9 'os' imported but unused
Change-Id: I0b775436268937ac19c4c5a1c3ba90dadf819ab2
2018-11-07 09:18:15 +00:00
Jimmy McCrory cf99208142 Fix issues with and enable Python 3 job
Python 3 disallows implicit relative imports.

When skip_tags is provided as a single string, it will need to be cast
to an array to avoid being treated as a list of characters when compared
to task.tags.

Change-Id: Ib8e7b67b9400fb983aecaae95f50bb3ec35d4c26
2018-10-22 15:54:24 -07:00
Jimmy McCrory 3eb9d79a1a Fix connection plugin for Ansible 2.6
inventory_hostname is not reliable to be used as container_name. When
delegating in 2.6+, the inventory_hostname of the delegating host is
passed to the connection plugin, additionally when a host doesn't have
its own container_name variable a cached inventory_hostname is used in
its place.

To get the connection plugin working with Ansible 2.6 this change
essentially allows the container_check to fall through when delegating
to physical hosts or hosts not in the inventory by:
- removing inventory_hostname as a fallback for container_name
- unsetting container_name when delegating to a known physical host
- unsetting container_name when delegating to a host not mentioned in
  the inventory

container_name can no longer be used as a group variable based on
inventory_hostname, it will need to be an inventory/host variable for
each individual host.

Also, when container_user is used the remote_tmp path needs to be set to
a system writable directory instead of the root user's home.

Change-Id: If2eb4c16273e19599f6ec3f0cba6b3573912c6a0
2018-10-18 13:27:27 -07:00
Markos Chandras f56d1242c9 strategy: linear: Inspect the delegated host on delegated tasks.
The purpose of the 'physical_host_addrs' variable was to create a
mapping with all the physical hosts and their IP addresses. However,
the function was broken because it used the 'physical_host' task
variable to obtain the name of the physical host for the running task.
As such, the dictionary only contained a single entry. This breaks
delegation between hosts, since there was no entry in the dictionary for
the delegated host. So if a task targetting containerA on hostA was
delegated to containerB on hostB, the IP address of the hostB was not
in the dictionary. The connection plugin code then used the 'hostB'
hostname to connect to the host instead of its IP. However, if there
is no central DNS to translate hostB to an IP or if the deployer
did not add such mapping to /etc/hosts we observed the following
problem.

fatal: [compute00]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh:
Could not resolve hostname controller00: Name or service not known\r\n", "unreachable": true}

In order to fix that, we now examine the delegated host and create the
appropriate mappings if the task is delegated. This ensures that the
physical_host_addrs variable also contains information for the delegated
hosts instead of just the original one.

Change-Id: Ic84ad3ccdd8283a4ab3fb3130d4d5e520ab736de
2018-08-31 10:06:31 +01:00
Logan V 3f224cdef7 Fix physical_host loading from group/host vars
Commit I76af71ffe19b2a8afb18ed731f1362a4e399c613 broke the ability
to define physical_host in host_vars and group_vars. The physical_host
is only loaded from the ini inventory.

Change-Id: I3798529cf2bf73afbff2d2ef0f3fe17d91123fbc
2018-07-12 18:13:18 -05:00
Kevin Carter 94c93e533f
Correct severe performance problem
When running our plugins we're seeing a severe performance impact in
the ansible run-time. As it turns out, within the linear strategy we're
looping over all physical hosts in an environment and pulling address
information out to store as a task variable. This action causes all
tasks to reload all {group,host}_vars for all hosts and that repetitive
action results in a massive slowdown, especially when there are many
hosts within the inventory.

Results from a normal OSA playbook run on 50 hosts:
  time openstack-ansible openstack-hosts-setup.yml --forks 15
  real    37m44.263s
  user    35m30.984s
  sys     2m23.252s

Results with the strategy plugin disabled on 50 hosts:
  time ANSIBLE_STRATEGY_PLUGINS=false openstack-ansible openstack-hosts-setup.yml --forks 15
  real    5m8.641s
  user    6m30.792s
  sys     1m32.512s

This change scopes the physical host look up to the task and target host
and adds conditionals around the process to ensure we're not forcing a lookup
that has no chance of succeeding.

Results with this patch applied on 50 hosts and no forks:
  time openstack-ansible openstack-hosts-setup.yml
  real    6m40.573s
  user    7m53.336s
  sys     1m25.828s

Should anything like this come up again debug statements have been added
to the method so that we can enable verbose mode and know what is being
parsed.

Change-Id: I76af71ffe19b2a8afb18ed731f1362a4e399c613
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-07-11 19:06:00 -05:00
Logan V a743d4c81f Template the physical host address
In the scenario where the fallback group 'all' is used to find the
physical host address, there are typically containers present in this
group with ansible_host settings like "{{ container_name }}".

When this happens, you get physical_host_addrs lists like:
aiotest1 127.0.0.1
test1 1.1.1.1
test-1-zbx1 {{ container_name }}
test-1-cdns1 {{ container_name }}
test-1-lb1 {{ container_name }}
test-1-lb2 {{ container_name }}
fatal: [aiotest1]: FAILED! => {"msg": "'container_name' is undefined"}

To avoid this we need to run the physical_host_addr through a templar
loaded with the physical_host_vars:

aiotest1 127.0.0.1
test1 1.1.1.1
test-1-zbx1 test-1-zbx1
test-1-cdns1 test-1-cdns1
test-1-lb1 test-1-lb1
test-1-lb2 test-1-lb2
test-1-tpxy1 test-1-tpxy1
ok: [aiotest1]

Change-Id: I7a4e89c579a3e688decaeb91b9035500129c7b16
2018-06-29 23:44:37 -05:00
Logan V 76bec48bc9 Update physical host mapping for ansible 2.4+
On ansible 2.4+, when 'ansible_host' is defined in the inventory
ini file, the physical host mapping in the strategy plugin works as
expected. However, if the ansible_host setting is in host_vars/<host>.yml,
then the mapping does not work.

Change-Id: I22c09c4de86233329f337b535b709cd5320c40d3
2018-06-26 06:30:59 +00:00
Jimmy McCrory 4a262efbb4 Fix connection plugin for Ansible 2.5
For 'physical_host_addrs' to be able to be looked up by the connection
plugin it will need to be included in the 'task_vars' dictionary which
is eventually passed to the task executor. Moreover, we remove the now
obsoleted[1] 'set_host_overrides' function.

If the 'container_name' variable is available, it should take precedence
over 'inventory_hostname' for determining a container's name.

The 'container_user' variable is now set through 'set_options' instead
of a host variable and, when it is available, the connection plugin's
shell is initialized with a system writable temp directory to avoid
requiring additional privileges for the user.

The container check has been updated to also check 'container_name'
against 'ansible_host'. If these match, the container is only known by
an IP address and will require a direct SSH connection.

[1]: 23b1dbacaf
Co-Authored-By: Markos Chandras <mchandras@suse.de>
Change-Id: I16bc76f17ae5d2da4803fd7cdcdcfe72e77f9df1
2018-06-22 17:05:30 -07:00
Paul Belanger f253c51d39
Add container_user to SSH connection plugin
This exposes the container_user setting to an operator and allows them
to override it. This is helpful when you want to use 2 different
username for your physical_host user and container user.  Today if the
physical_host user is root, the container user will always be root
too.

Change-Id: Ie8e32b0441a5d72fcbc3cbf9769d8a7dc7548f08
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-03-19 11:35:11 -04:00
Jimmy McCrory d8e4236b5e Make connection plugin compatible with Ansible 2.5
Previously, the MAGIC_VARIABLE_MAPPING constant was updated to make
additional OSA required hostvars available to the play_context. In 2.5,
MAGIC_VARIABLE_MAPPING has a different location but updating it is no
longer necessary. Connection plugins are now able to access any hostvars
and options defined within their docstrings. The upstream connection/ssh
plugin's docstring has been copied here with the addition of OSA's
required options.

Change-Id: I6b1bcd4ec5e9d2701bd26e24e0f7645629ae4f3d
2018-02-20 09:43:51 -08:00
Kevin Carter 7d973364c5 Add basic plugin support for nspawn type containers
This change modifies our connection plugins to allow us to use nspawn
type containers. The change will pull in the container_type variable and
set the padding for commands accordingly.

While the module has been setup to default to LXC which ensures we've no
conflicts with existing deployments, the container_check function has been
set to ignore "nspawn" type containers until we have a suitable/reliable
method to connect to them without envoking ssh.

Change-Id: I8877c244af6458ca9840ea0519ab457a726feefd
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2017-12-01 04:33:18 +00:00
Jimmy McCrory 5cd56106c4 Make linear strategy plugin 2.4 compatible
In Ansible 2.4, the InventoryManager 'get_group_dict' function was
renamed to 'get_groups_dict'

Change-Id: Ic758253af7b2fe742d253f363edeb2953130a2f5
2017-09-12 11:19:47 -06:00
Logan V 6c6753bbed Fix the physical_host_addr vars lookup
The inventory lookup to determine the map of physical_host ->
ansible_host vars was never locating ansible_host, and instead
always defaulted to physical_host's inventory_hostname.

This fixes the lookup so the inventory_hostname will only be used
as a fallback if ansible_host is not set on the physical_host.

Change-Id: Ibbadbaf2a6a5b7323ee6587da3773b8459fa2eb6
2017-07-05 09:09:59 -05:00
Jimmy McCrory 8b1a07d2d6 Fix delegation to containers between hosts
Tasks which delegate from a container to a container on a separate host
currently fail because the 'physical_host_addr' variable, which those
tasks use to connect to a physical host, is exclusively set as the
address of the originating container's physical host.

Assign a new host variable, 'physical_host_addrs', which contains a
mapping of each physical host and its address, so the tasks can now
lookup of the correct address to use depending on the target container.

Change-Id: If594914df53efacc6d5bba148f4f46280f5a117d
2017-06-19 16:14:47 -07:00
Jimmy McCrory 8f8d9cfa11 Connect using physical host's ansible_host var
Within the strategy plugin set a host var, 'physical_host_addr', which
corresponds to the 'ansible_host' var of the host's 'physical_host'.

Use this within the connection plugin rather than the hostname of the
physical host, which may not exist in DNS or the deployment host's
hosts file.

Closes-Bug: 1695944
Change-Id: I75f9d0f55ecd875caa1bf608a77c92f950b679a1
2017-06-07 21:46:23 -07:00
Logan V d8c0341ab8 Support delegation in strategy plugin
The strategy plugin previously did not support container detection
when the delegate_to host was a container. This adds support for
delegating tasks to other containers, so the strategy will perform
the same container vars lookups so the SSH connection is toward the
physical host of the container, and the command is wrapped with
lxc-attach based on the container_name.

Co-Authored-By: Jimmy McCrory <jimmy.mccrory@gmail.com>
Change-Id: I56d8afddbccf01f2944d2fdd505b601a4b048374
2017-04-21 23:52:09 +00:00
Logan V 5d6c040d18 Implement tag filter strategy
Renames the "artifacting" strategy to "tagfilter" and implements
a more generic tag skipping implementation that can be defined
from within the ansible vars structure dynamically on a per-host,
per-group, or per-playbook basis.

To use the tagfilter strategy, run the playbook with
strategy: tagfilter

and define a hostvar named "skip_tags". Skip tags can be a string
or a list, and any tag CONTAINING a skip tag (an exact match is not
needed) will be skipped.

Also added is a check for a var called "skip_handlers", defaulted
to True. When skip_handlers is enabled, no handlers will be run
regardless of whether the task is skipped or not. If set to False,
handlers will be allowed to run as normal except when tasks are
skipped by the skip_tags.

Change-Id: I649707e6744e03763c4e786d203716ebf657ab48
2017-02-24 09:35:48 -06:00
Jean-Philippe Evrard 655d777734 Add a new strategy filtering tasks
Ideally Ansible should allow (from a playbook) to run only part
of a role thanks to its tags (like the --skip-tags or --tags from
the cli). But it's not possible now.

This code is a PoC code for introducing a kind of filtering,
which basically skips tasks containing the word "config".

If you are building lxc images ("artifacts") of a role, you
probably don't want to run all the tasks in the role, and don't
want to run the handler tasks (because they start software for
example). This also addresses it.

To use this work, just include strategy: artifacting in your
play.

Change-Id: I006bc640c6563c959ceb835ddf5bef8d25dd7517
2017-02-22 19:38:29 +00:00
Jesse Pretorius 5af635977e Add ability to execute against a remote chroot
In some situations it may be useful to execute a role or
set of tasks against a remote chroot.

This patch aims to add that capability.

Change-Id: Icebec389807aa8782f97a95f63687eb5606f7b22
2016-11-22 12:28:35 +00:00
Kevin Carter e566c20b24 Remove deprecated ansible_ssh_host variable
This changes 'ansible_ssh_host' to 'ansible_host'. The 'ansible_ssh_host'
variable has been deprecated as noted here: [0].

[0] - http://docs.ansible.com/ansible/intro_inventory.html#hosts-and-groups

Change-Id: I39892435950eca6da773dd947894da2125eab147
Related-Bug: #1636606
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-10-25 19:43:47 +00:00
Logan V fcf7fc3735 Accept ansible_host in linear strategy
The physical_host detection in the linear strategy required
ansible_ssh_host be set on the physical host. Since ansible_ssh_host
is now deprecated, we will accept both the new var ansible_host and
the old var ansible_ssh_host.

Change-Id: I3504c1901cf8f9f6f60c5465b55269997955635e
2016-10-14 19:40:03 -05:00
Kevin Carter ffb02cfb55
Fix assumption that the main OSA inventory will be used
The role base test use a much simplified inventory which this plugin
was not accounting for. This resolves the assumption.

Change-Id: I995b62cbf825f15a93e3bb44f48b3527c9e5ba48
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-08-25 23:20:04 -05:00
Kevin Carter cb01efef66 Implement an opportunistic strategy and connection plugin
This change is creating an opportunistic Ansible execution strategy and
an update the ssh connection plugin so that it supports direct access to lxc
containers within ever having to ssh into them.

The intention of this change is to speed up execution time and reliability by
tuning the execution environment within Ansible to run faster while also attempting
to subvert transient ssh issues.

Change-Id: Ide34513bf82523257bdd2a8a68dff165f9927c56
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2016-08-25 17:45:10 +00:00