Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
Rename fixtures directory to job_fixtures to prepare for adding view_fixtures
directory in the following commits.
Change-Id: Ic20997cae020b542ddc22bf444fa6b92fbcae064
Setting desired version of black to 22.3.0 prevents issues with
the click dependency described in the #1966879.
Introduction of black 22.3.0 forced style changes in following files:
- doc/source/conf.py
- jenkins_jobs/local_yaml.py
- tests/base.py
These changes do not modify behavior of the programs concerned.
Signed-off-by: Jiri Podivin <jpodivin@redhat.com>
Change-Id: I42a46f9ed31ea0f404d698eec73d30975e2e6f4c
Before this change using an anchor/alias in !j2-yaml resulted in an
exception, even if retain_anchors was set to True:
> File "(...)/jenkins-job-builder/.tox/py38/lib/python3.8/site-packages/yaml/composer.py", line 68, in compose_node
> raise ComposerError(None, None, "found undefined alias %r"
> yaml.composer.ComposerError: found undefined alias '<alias-name-xxx>'
The reason was that LateYamlLoader.get_object_to_format() was calling
load() without retain_anchors=True so effectively it was resetting
anchors during deep formatting...
From now on yaml.load() is called directly to avoid unwanted
side-effects of load() which is meant to be called from other modules,
typically by parser.YamlParser().
Change-Id: I0fde43c5c27ddf78f18dc244d8dba5bd768306b7
The properties of 'project' objects are deepcopied in
YamlParser._expandYamlForTemplateJob(), including any Jinja2YamlLoader
objects produced by !j2-yaml tag. Because of the missing __deepcopy__()
definition in Jinja2YamlLoader, the parent class' implementation was
used, causing the deepcopied object be of Jinja2Loader type.
Consequently the rendered value was always a string, not an instance of
LateYamlLoader as it supposed to be.
This change fixes the issue in Jinja2YamlLoader and potentially in
LateYamlLoader, however the latter is currently not deepcopied anywhere
so no test could be written to cover it.
Change-Id: I24fda368a4af0e9aac7b78c478ac7b4eddf27b9b
The tag provides Jinja templating capabilities for generating parts of
YAML structures. Two potential use cases are provided as test cases and
are linked in the documentation.
The new tag should also help address some use cases people were asking
about, like here:
https://groups.google.com/g/jenkins-job-builder/c/HkVZVuBDlKM.
Change-Id: I96392e42c3c79a9be0a8f736506908701251dd62
When running JJB under py3 (py3.6) it started to fail with exception
TypeError: cannot serialize '_io.BufferedReader' object
This is happens when doing deepcopy of a LazyLoader object which
contains an instance of BufferedReader object.
Under py2 this works but under py3 it is not.
This commit adds custom __deepcopy__ method to LazyLoader that
avoids deepcopy of a self._loader.
Change-Id: I00844fc06ebf84538d557c33f1179a26cb18202f
Story: 2007227
Jinja templates are not deep-copyable so they cannot be used in
"defautls" sections or to pick defualts for job groups or projects.
This works around the issue by waiting until we render the template to
construct the template itself.
Story: 2006431
Task: 36337
Change-Id: Ief31fdaac06bb14d0aaba71c8c0e658a7f861671
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.
Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
This also adds support for using custom loaders in job definitions
not just job-templates and builders. I have also added unit tests for include-jinja2-as-yaml custom loader.
Example usage:
- axis:
type: user-defined
name: VERSIONS
values:
!include-jinja2-as-yaml: versions.j2.yaml.inc
Where versions.j2.yaml.inc is
{% for possible_versions in configuration["possible_versions"] %}
- {{ possible_versions["versions"] }}
{% endfor -%}
Change-Id: I15a431d5a86b29d704efda8576965ade2b7dcd2f
In cases when there are many j2 templates and some of them use
include tags, correct jinja loader is overriden with one of loaders
related to other templates.
This patch fixes it with workaround. Which is overwrite incorrect
loader with initial.
Change-Id: Iec21c24fe6e4bfedf281d7215e520298381f07a7
- upgrades hacking to current version
- sorts new linting issues
- sorts bug with sys.reload on py3
Change-Id: I4a18abc93116667a2733e8aec619ac59ea73d630
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
If set to True, YAML anchors can be referenced across files, allowing jobs to be
composed from bits of YAML defined in separate files. False by default.
Story: 2000338
Task: 2547
Change-Id: I034ce3bce0030093cb8d4266dabbdb06d96306d6
Adds a yaml application tag '!join:' to support defining data as a list
while having the resulting contents automatically joined together with
the specified delimiter and passed as a string into the resulting object
returned by yaml for JJB to process.
This allows users to store long lists of data that is combined together
for use with Jenkins plugins that expect a delimited string of arguments
instead of needing each module in JJB to provide individual support.
Change-Id: I745181ade3926d5c29708963189ae499a0378ece
This allows the use of the {% include %} tag[0] within Jinja2 templates,
with the same search path used for looking up other templates (i.e. the
one which LocalLoader is instantiated with).
[0] http://jinja.pocoo.org/docs/2.9/templates/#include
Change-Id: I0403c385ee317e1460a80a9bf717aa4712f37e2f
File inclusion is debug info, info level would
spam console and make user unable to get
progress info without it.
File inclusion can easily mean >100 lines for
users relying on templating.
Change-Id: I013d86d8563fd75f5255578c0c735c4f2d5c074b
Introduce the registry.MacroRegistry class to handle:
* registration of macro types via setuptools' entrypoints
* registration of individual macros for lookup by component list type
* expansion of macros references during YAML "parsing"
As a consequence there is a reduction in performance due to moving the
expansion of macros from inline with XML generation, to requiring
multiple passes over macro component lists.
This decrease in efficiency results in approx ~30-50% increase in unit
test time. Since this will allow for jobs to be expanded from
templates/macros in parallel with future changes, it is a reasonable
short term trade-off as the most computationally expensive task is
updating the definitions on the remote master
Change-Id: I292c6b1f8472370282205426cd8ceb847eb969bd
This template included using !include-jinja2:
"""
{{ my_var }}
"""
is rendered the same as an existing template that looks like this:
"""
{my_var}
"""
This also allows the use of Jinja2's richer syntax:
"""
{% for test_environment in configuration.get("envs", ["py35"]) %}
tox -e {{ test_environment }}
{% endfor %}
"""
Story: 2001135
Change-Id: Ia3ee21822d6e9237f5ea46796bc8810ecac61e2c
Output the variable inputs used that trigger an error when expanding
template names to the error logger channel in a sensible format.
Ensures that when indented variable inputs for templates result in
exceptions when expanding a template name, that the project, template
name and variables that failed to be iterated over are outputted in a
log error message along with the original set of inputs from the
project definition to make it easier for end users to find where the
error has been made in a JJB definition.
Add code to allow dumping of variables stored in OrderedDict
transparently to match the input format used in JJB definitions and
hide the implementation detail of using OrderedDict to be within the
localyaml library.
Change-Id: I660bb0ca3b109e1a861948d6a867f185047b90ae
To allow filenames referenced by the application specific yaml tags to
contain template job variables, use a lazy loading object that provides
a format method that can be called by the deep_format function.
Instead of processing the file, when a KeyError occurs on attempting to
call format on the filename after the yaml tag, create a LazyLoader
instance to wrap the data and provide a format method that can be called
at a later stage.
In order to call the correct method on the original Loader class,
LazyLoader needs to be given the custom tag class, a reference to the
loader and the node object. Using the tag class it can call the
from_yaml() method with the loader and node object to return the file
contents.
Since the result from the LazyLoader instance is triggered by calling
the format method, there is no need to escape the brackets used by
pythons format method since the output will not be passed through it.
In order to ensure this behaviour, nodes passed to the method handling
the '!include-raw-escape:' tag class, which need to use the LazyLoader
approach will convert to the '!include-raw:' tag class to the
LazyLoader initialization instead.
Due to a bug in sphinx with use of 'note' admonitions and manpage
generation, need to update to a version >= 1.2.1.
Change-Id: I187eb83ba54740c2c1b627bc99c2d9769687fbc7
Story: 2000522
python 2.6 is no longer supported and adding a marker
to requirements.txt for installing ordereddict only in py2.6
envs was not an option because markers are not supported in
older pip versions.
So remove python 2.6 support completly.
Change-Id: Iebdd999b469c9a9681b0d7e9f50cc488a8820953
Implement custom tags for including files by subclassing YAMLObject to
allow PyYAML to easily convert from the Yaml nodes.
Change to support both lists and strings under the same tag name and
deprecate the old tag naming. Rename tests using the deprecated tags to
ensure they are not used for examples. Remove old tests that simply
duplicate others.
Change-Id: I1a8d3376ea6e4918a2adb05fb856b939a3124d74
Use io.open() to allow reading and writing of files in 'utf-8' format
irrespective of the terminal encoding selected.
Change-Id: Ie952617a34c0719efc59a7729d698beafaa477b0
In code the `include_path` and defaults like `.` added to a set as opposed a list.
Resulting in that the order is machine/hashcode specific and
that user cannot make i.e. `.` to have higher precedence than
the fallbacks.
Without doing this you have to use absolute or unique paths which
might not always be possible in your .ini file.
Change-Id: I9721b168a67e799dfc19d7df951b99c0749b52e0
Anchors and aliases were expanded internally within JJB's yaml loading
calls so they were limited to individual documents. Now, included files
will have access to aliases of anchors already defined at previously
processed files.
Example:
- default:
name: default-timeout-wrapper
timeout: &timeout
fail: true
elastic-percentage: 150
elastic-default-timeout: 90
type: elastic
- wrapper: !include include002_1.yaml.inc
Previously was not possible to use '*timeout' alias inside
include002_1.yaml.inc file
Closes-Story: 2000173
Change-Id: Ic031ddbb0310bd11748183fbde9502735c3b7169
Add variants of the include-raw* tags, which accept a list of files.
Those files are concatenated and included as a string data into the
calling yaml construct.
Change-Id: I6af87a298268acc8a73d7a2b50f9f99733d8723a
Convert to use idioms that work for both python 3 and python 2.6+ and
ensure that a suitable version of dependencies is included for python 3
compatibility.
Update python-jenkins to 0.3.4 as the earliest version that supports
python 3 without any known regressions. Add an extra parser check for
missing 'command' due to changes in how argparse works under python 3.
To access the first element of a dict in both python 2 and 3,
'next(iter(dict.items()))' is used as the standard idiom to replace
'dict.items()[0]' as 'items()' returns an iterator in python 3 which
cannot be indexed. Using 'next(iter(..))' allows for both lists and
iterators to be passed in without unnecessary conversion of iterators to
lists which would be true of 'list(dict.items())[0]'.
Original change which was reverted due to breaking use of job-groups is
If4b35e2ceee8239379700e22eb79a3eaa04d6f0f. This replaces the previous
conversion of 'dict.items()[0]' to 'dict.popitem()', which would result
in removing a job-group when first called, thus defeating the benefit of
being able to reference the group mulitple times. This usage has been
replaced with 'next(iter(dict.items()))' as a non-modifying alternative
that still avoids creating unnecessary copies of data while working for
all supported versions of python.
Change-Id: I37e3b67c043dadddb54e16ee584bde3f79e6a770
This reverts commit 1d7647fa85.
This change altered the xml output (by forcing it to fail) which a
backward compatible change should not do. Revert it in order to get this
compat change in without breaking that output and test.
Change-Id: I20f66fb1bd9c70a0debbdd5eebacf6ec5d0f5df9
Convert to use idioms that work for both python 3 and python 2.6+ and
ensure that a suitable version of dependencies is included for python 3
compatibility.
Update python-jenkins to 0.3.3 as the earliest version that supports
python 3 without any known regressions. Add an extra parser check for
missing 'command' due to changes in how argparse works under python 3.
Where contents should be retained, to access the first element of a dict
in both python 2 and 3, 'next(iter(dict.items()))' is used as the
standard idiom to replace 'dict.items()[0]' as 'items()' returns an
iterator in python 3 which cannot be indexed. Using 'next(iter(..))'
allows for both lists and iterators to be passed in without unnecessary
conversion of iterators to lists which would be true of
'list(dict.items())[0]'.
Alternatively, where further access to the data is not required,
'dict.popitem()' is used.
Change-Id: If4b35e2ceee8239379700e22eb79a3eaa04d6f0f
Python 3 enables hash randomization by default, additionally tox 1.7
turns on the same randomization for earlier versions of python by
default. Need to ensure that order of iteration over the yaml data and
resulting XML has deterministic order for testing.
Adapts https://gist.github.com/enaeseth/844388 which ensures data read
by yaml will have its order retained in a predictable manner across
multiple python versions.
Additionally it seems more sensible to ensure that the order of
generated XML snippets corresponding to the input yaml files are
consistently in the same order as the entries in the source files.
Closes-Bug: #1333349
Change-Id: I6bf6d298a2609cc6ddbbc6b02b7f1a04413a5c89
JJB didn't actually handle unicode data very well for a couple reasons.
First the local yaml loader was loading files into yaml as strings
instead of unicode which we should just go ahead and do because yaml's
built int loader loads utf-8 by default (and we don't override the
default). Second we need to do parameter substitution on unicode and
regular strings so change the substitution typecheck to use basestring
instead of str. Finally we need to use UTF-8 as the encoding when
emitting XML so do that.
Add tests to actually test this in the yamlparser tests. The addition of
these new tests comes with a little bit of cleanup in the test classes
to make sure we load unicode files as utf8 more consistently.
Change-Id: I2169e19aae2cdc7ddbd1e7217ef7584c786a039a
Fixes-bug: 1361090
Add support for local tags which are application specific to allow
including of other yaml files or code from scripts. Allows for code to
be maintained and tested as seperate files, as well as reduces
duplication of yaml code that cannot be macro'ed or easily templated by
including it from a common file.
Adds support for the following tags:
'include' - load file as yaml code
'include-raw' - load file as data (for scripts)
'include-raw-escaped' - load file as data with escaping braces '{}'
as default for use with job-templates
Use configuration file options to provide a search path for the files.
- Test behaviour of yaml tags independent of any XML generation
by comparing json result of yaml parsing to verify that certain
tags do/don't recall the yaml.load() method.
- Add examples for the include tags via addition tests for YamlParser
class
Inspired by
http://stackoverflow.com/questions/528281/how-can-i-include-an-yaml-file-inside-another
Change-Id: Ib90a07043112d4739d6529ceddbc9817668bcec0
Vsevolod Fedorov