At the moment we do resolve template 2 times for some historical reasons
This leads to {% raw %} tag as well as any other content be not respected
on the second renderring, since after the first one these tags
are gone, so no escaping or whatsoever work.
At the same time there're no obvious reasons to render second time,
since all override content is being renderred already. So there should
not be anything left that indeed needs this second run.
Change-Id: I414b1b3b4e426316707dda23b9aeff6b6f4f7c2f
Closes-Bug: #1649381
In case template contains some Jinja content that we don't want to
resolve, there're not much options due to the bug, that ignores
{% raw %} tags. The other way to work it around is to override tags that
Jinja will use to detect the content it needs to render. This way
content that should be renderred can be placed in the same template
with the one that needs to stay intact.
Change-Id: I4d28f5ab03858f5555f5e9e555da1dddc0f2b016
Related-Bug: #1649381
In order to pass functional tests on newer ansible we need to replace
dropped inlcude with proper import_playbook module.
Change-Id: I2bf9ca6003b66d5f7b8be9d7b54e12cb9b56f3c6
Use localhost and container1 as proper targets for the test playbook
rather than a combination of import_tasks/delegate_to for container1.
The previous code did not run the imported tasks against container1,
but instead a second time against localhost. This makes the handlers
checking the output from ansible diff fail as on the second run
against localhost there are no changes, giving an empty diff.
Changing the test play targets to localhost:container1 makes it more
obvious that the whole test playbook should be run against the host
and the container. It is not clear from the old code if the original
intent was to specifically test config_template tasks delegated from
the host to a container.
Change-Id: Ibc72868d9957567fe86a9d3205a80ec33441baff
In cases, when key already exist, but commented out without space
(ie "#key) and is not in default section, it's not updated with
sectdict.insert statement. That is possible happens because
self._sections keys are protected and can't be overwritten (at least
new assignment makes no difference)
Change-Id: Ie0c588a2d559bd38c68b3d018e547a036d83214c
Closes-Bug: #1889547
Sometimes there might be a key in configfile, that should be empty but
defined correctly. This patch fixes such behaviour and adds testing.
Change-Id: I693bc3a2aade7015d0465583001962df4ba53443
This change will allow config_template to parse list only files
and extend or replace the content of the list based on the
selected module options.
Change-Id: I2a24033b0323bcc25bd1b50ffb4034441ab2f468
Signed-off-by: Kevin Carter <kecarter@redhat.com>
The config_template comment parser will now respect all comments in
INI files, as they are written. This will ensure spacing, paragraphs,
and other comments bits that may be in an INI file remain intact,
even when overriding options and sections. With this feature we'll
now be able to insert options in OSLO config generated files without
truncating or making a mess of the file structure.
This is an internal enhancement and requires no change from the
operator or from within any ansible task. To ensure enhanced comments
are working, new tests have been added which will run though all of
the `config_template` INI file functions using a mock service file.
Existing tests for the old comment structure has been removed.
This includes tests that were running redundant tasks for file
diffs or expected the only style, merged, comment layout. All of the
tests have been broken out into descriptive task files. This was
largely done for readability. It was difficult to see what tests
we had and how I needed to extend them to test the enhanced comments
functionality given the INI type tests were all thoughout the
`tests/test-common-tasks.yml`. Now that the files have been broken
out developers will easily be able to audit our tests making it
simple to extend.
Change-Id: Ia6cdc215c35fa9ac45b718c211616a9887a74e37
Signed-off-by: Kevin Carter <kecarter@redhat.com>
The config template action plugin assumes that the local node is
supplying a configuration file that will be used to copy content
to a target node. This change gives the module the ability to
change configuration files on target nodes using all remote
sources. The new boolean option, `remote_src` has been added to
enable or disable this functionality. When enabled, the module
will retrieve content from the target node and load it as
"user_content". By pre-loading it in the user content the
module will function normally without any major structural
changes or performance impact.
Change-Id: Id9c7e16fb935c2da0b32b7cf53449f68bd1e9c89
Signed-off-by: Kevin Carter <kecarter@redhat.com>
Added a flag called yml_multilines which when used will not split on
lines that have a newline UNLESS they have a comma.
Change-Id: I90364403e215a67b320dfc7e67a85d47c774e634
Partial-Bug: #1819974
Python 2.7 ConfigParser does not accept whitespace before comments.
To keep config files produced by ansible-config_temple parsable
we will strip out the whitespace preceding comments.
Change-Id: I39c88578d0a9e42fded7410d7eb4ec8bd415550c
Related-Bug: #1755821
Incorporate the musltistropt test cases from openstack-ansible-plugins
/tests/test-config_template.yml excluded from ansible-config_template
Change-Id: I57c4eb3e1c879671facfa4564f78869961a4e9f2
Closes-Bug: #1815620
The connection plugin no longer falls back on using the
inventory_hostname as the container_name. Set container_name as a host
var for each container in the test inventory.
Change-Id: I60e24d1cd02dbb96b96325df4eb483eef0b6df83
Compare dict vars of before and after configuration to determine whether
the config keys or values have changed so a configuration file will not
be incorrectly marked as changed when only the ordering has changed.
Set diff return variable to a dict of changes applied.
Change-Id: Ie67119b1420936c8ed89f8338ea9dce4c47e185c
With the more recent versions of ansible, we should now use
"is" instead of the "|" sign for the tests.
This should fix it.
Change-Id: Ifa4bfaf645621ef23fa8f9d7893b4f596052f655
The default section in an ini file will always be 'DEFAULT', but for some
configuration files you may want to be able to set this to ensure
another section is at the top. 'global' for example, when working with
fio.
This PR adds the ability to set the section to be at the top, but will
default to 'DEFAULT' so no change will happen to existing tasks running
config_template.
Change-Id: Ifbc0b91aef46f3c2d98e73fdc9ab888244550bab
unicode is nolonger a type in py3 - instead the str type is used for
unicode, and the bytes type would cover the str type from py2.
To avoid having to version the code, we should try to use unicode,
except the NameError and continue as though it is py3 if that fails.
Additionally, this patch adds a test that will fail if you revert the
config_template.py back to it's original in py3.
Closes-bug: 1763422
Change-Id: Ifda972caada27ade2d80f77b3df70568406226ff
The lint tests make use of the common linters test
used by the roles, so we need to ensure that the
run_tests.sh script is updated and the configuration
adjusted accordingly.
Change-Id: Ic623e6074a85e08416b30e3e42c48fe8873113d6
As part of the ceph-ansible project, this change was put in:
ec042219e6
This was to handle the case where the config would change, initiating a
restart handler, even though no actual change happened.
We can juse use sorted() since we are passing back a list with .items()
which can then be sorted.
The above change didn't include a change to ensure the order of the
sections, which meant that although the individual items were sorted the
sections could change order causing unnecessary changes.
Additionally, this adds a test to retemplate the file 3 times, with a
lot more sections to ensure the order remains the same, this will fail
when the config template changes even though no changes have happened.
Change-Id: I7c7cd9ef1dd767a2a5d754ac4663cf8655dd60c2
(cherry-picked from plugins repository)
config_template currently strips comments out from the INI files. This
commit adds support for comments.
The default ConfigParser supports reading comments but it doesn't writes
them back. There are other libraries that have better support for
comments, however, adopting another library would require a bigger
refactor of this code.
Change-Id: I376be69cd2cf34dc147f9ade716791125a424e2c
With the merge of https://review.openstack.org/520177 in the
tests repo some ansible-lint failures which previously were
not being picked up are now detected.
This adds the appropriate skip tags to the tasks so that they
are not evaluated by ansible-lint.
Change-Id: I820aa77ac097d4f3d043fb434255923847e3d650
This patch implements an initial set of jobs intended to match
the current job execution method. It does not intend to improve
how the jobs are executed - only to replicate what is currently
in openstack-infra/openstack-zuul-jobs and provide the platform
to iterate on.
Change-Id: Id5a41c4f09bfc63b373e83493dd85a53850bd71c
- Test the container awareness of the connection plugin (the lxc-attach
execution functionality when container_name is set on a host).
- Test delegation to containers
Depends-On: Idb186a061a396713a0e449d47ec262b30d5006ca
Change-Id: I2331d5d5c9cf3d4f8fc83639f6047092232f5887
Using AnsibleDumper adds support for dumping the internal ansible
data types that are used in hostvars.
Fixes-Bug: #1707265
Change-Id: Ie59bbef31306cd3e99aa827db875922e17084e83
When accessing nested facts, such as in ansible_local sections, guard
statements in the when clause can get quite long, due to having to
repeate the 'in/not in' logic for every key at every level.
The get_nested filter attempts to instead to move the guard into a
single line, making the conditions easier to write and maintain.
As example,
('openstack_ansible' not in ansible_local or
'swift' not in ansible_local['openstack_ansible'] or
'venv_tag' not in ansible_local['openstack_ansible']['swift'] or
ansible_local['openstack_ansible']['swift']['venv_tag'] == swift_venv_tag)
could be rewritten as
get_nested(ansible_local, 'openstack_ansible.swift.venv_tag') == swift_venv_tag
Change-Id: I3b43c25c8783c43cf5285f2b3e7267b2c5712ea0
It is sometimes useful to tell ConfigTemplateParser to write
out options that are valueless and not suffixed with '=' or ':',
such as when overriding a my.cnf.
This commit adds the 'ignore_none_type' attribute to the config_template
module. If this attribute is set to false, then valueless options will be
written out to the resultant INI file as-is, without the '=' or ':' suffix.
Change-Id: I5c88b2019c01b44193a5d0df9299ecce6de52f01
Partial-Bug: #1693234
When users specificy the 'content' attribute when
using the config_template module, it's possible that
this input can be a dict. This commit handles
dictionary input, and dumps it accordingly based
on 'config_type'
Change-Id: I1f12810ad7ce5746f8938ec9d608c532ba530d58
When executing the tests repo clone in OpenStack-CI,
use zuul-cloner instead of git to enable cross-repo
testing. This ensures that if a dependent patch from
the tests repo is noted using 'Depends-On: <change-id>'
in the commit message, that patch will be included.
Depends-On: Idce7abebf32f24c356a27e099fbca954d917402b
Depends-On: I5da7802d61d2ab6b03908138e3a3ed2db22e3d29
Change-Id: Ib9e5f4c93529b6bf74e37ce9f2feb0faa87a3b81
This is a lookup in case you need to parse a ubuntu repo Packages file
for a list of packages. It's useful in osa if you want to have a mirror
of an ubuntu repo, and output a list of its packages, with their
location and checksum.
You can use the output dict to feed another tool (for example
downloading with aria).
Change-Id: I10ba72f0b64267ae6afe57317e135d782445e934
Renames the "artifacting" strategy to "tagfilter" and implements
a more generic tag skipping implementation that can be defined
from within the ansible vars structure dynamically on a per-host,
per-group, or per-playbook basis.
To use the tagfilter strategy, run the playbook with
strategy: tagfilter
and define a hostvar named "skip_tags". Skip tags can be a string
or a list, and any tag CONTAINING a skip tag (an exact match is not
needed) will be skipped.
Also added is a check for a var called "skip_handlers", defaulted
to True. When skip_handlers is enabled, no handlers will be run
regardless of whether the task is skipped or not. If set to False,
handlers will be allowed to run as normal except when tasks are
skipped by the skip_tags.
Change-Id: I649707e6744e03763c4e786d203716ebf657ab48
The default py yaml `safe_dumps` keeps to the yaml spec which states
incremental indentations are not required. Sadly, without the
indentations some consumers of yaml configurations files will break.
This commit adds an incremental indentation dumper to cater for all
yaml configuration consumers.
Change-Id: Ifa5bcc6ff9f0a9cb5e619e62d0356f5d6887ca6e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Copy module can take content argument, I think we should introduce this
feature to config_template. It brings the possibility to use lookups for
inline passing of content to template.
Example, we could use:
config_template:
content: "{{ lookup('file',<file>) }}" to replace current behavior
or:
config_template:
content: "{{ lookup('url','<url>',wantlist=True) | join ('\n') }}"
to bring inline templating of external sources.
Change-Id: Id5b2743d309f0313603afbbf84279ce0b1e49cfb
Signed-off-by: Jean-Philippe Evrard <jean-philippe.evrard@rackspace.co.uk>
This patch consumes the test scripts implemented by
https://review.openstack.org/375061 to ensure that
the tests and test preparation is consistent and
more maintainable.
Change-Id: Iffbb70a29d1c5c0a7a9d0956b8df39fa53e0f686
By default a list item in a JSON or YAML format will extend if a its already
defined in the target template and a config_override using a list is being set
for the existing "key".
This change allows that functionality to be toggled on or off. The boolean
option to enable list extension is ``list_extend``.
To maintain the API this feature will remain True by default.
Co-Authored-By: Travis Truman <travis_truman@cable.comcast.com>
Change-Id: I97e06aef2cc778f048f3d6863fe61d10eddb8602
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
Add a playbook to test each of the osa-filters.
The linters tox test environment has been updated to include a run of
this playbook.
Change-Id: I0bd877ce7bbd56edc52cf071d02d55d0210f9829
Dmitriy Rabotyagov