Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
The tag provides Jinja templating capabilities for generating parts of
YAML structures. Two potential use cases are provided as test cases and
are linked in the documentation.
The new tag should also help address some use cases people were asking
about, like here:
https://groups.google.com/g/jenkins-job-builder/c/HkVZVuBDlKM.
Change-Id: I96392e42c3c79a9be0a8f736506908701251dd62
Resolves flake8 BLK related issues.
"BLK100 Black would make changes."
"BLK999 Unexpected exception: ..."
Change-Id: Ic6d4265e3d02b2f2a15ab851eaed98288c62730c
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Ensure that we can resolve any lazily loaded templates that might be
dictionary keys in yaml. For more informmation please see the added test
case in tests/yamlparser/fixtures/jinja-string04.yaml.
Change-Id: I67fbca422f2165729af15e4d039278ad66f6240b
Story: 2006431
Task: 36338
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.
Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
This also adds support for using custom loaders in job definitions
not just job-templates and builders. I have also added unit tests for include-jinja2-as-yaml custom loader.
Example usage:
- axis:
type: user-defined
name: VERSIONS
values:
!include-jinja2-as-yaml: versions.j2.yaml.inc
Where versions.j2.yaml.inc is
{% for possible_versions in configuration["possible_versions"] %}
- {{ possible_versions["versions"] }}
{% endfor -%}
Change-Id: I15a431d5a86b29d704efda8576965ade2b7dcd2f
- upgrade flake8 to 3.6.0
- remove use of hacking since pre-commit replaced it
- mentioned minimal flake8 version on pre-commit config
- fixed newer linting errors
- disable W504 as it seems not possible to fix (conflict with W403)
- replaced buggy tox deps installation with native pip install, which
address the issue of missing to install new deps when the reqs files
are updated, developer being forced to to recreate the virtualenv.
- prepare for migration from tox-pep8 to tox-linters, now these being
just aliases.
Change-Id: I310578dce215aaf00a5b2d54716f90da9a1ecb4d
Depends-On: https://review.openstack.org/#/c/613726/
It can be confusing when adding variables to templates and macros to
discover that '-' are not valid characters in the key name.
Change-Id: I8b10371ddac54c19a92a9d0811cd549a492b3dbb
Negative values can't be set as default values
currently, for example:
num-to-keep: '{num_to_keep|-1}'
It will return an exception saying: 'parameter missing
to format'.
This should also allow other patterns as long they do
not contain the closing curly bracket.
Change-Id: I26dafb80b120f2c071dda6d9a27319d533b0f874
Make use of an improved regex pattern matching to simply escape the
template name when passing into the params to allow the name be
preserved and used as input to other params.
This avoids needing to perform an additional branch test against all
variables being formatted.
Change-Id: I9c0ac8996d520b9acff3d29039c5c3d76f56d899
Depends-On: I8a74f9b4236ca7bcc72dd207fca23c2bf6a7c801
Before:
{{var}} -> {var}, but the regex matches at the inner brace, but
re_replace returns the whole match (match.group(0) = "{var}") unchanged, passing
"{{var}}" to Formatter.
{{defined_var|def}} -> {{defined_var}}, because the regex matches at the
inner brace and re_replace returns "{%s}" % key ("{defined_var}"), passing just
"{{defined_var}}" to Formatter.
{{undefined_var|def}} -> exception, because the regex again matches at
the inner brace and re_replace returns default ("def"), passing "{def}"
to Formatter which then fails as def is not a defined variable (assuming
it isn't and allow_empty = False).
{{undefined_var|defined_var}} -> value of defined_var, same as above,
"{defined_var}" is passed to Formatter.
By preventing the regex from matching at inner braces, none of these
weird cases happen and even-braced strings are passed to Formatter
unmodified.
Change-Id: I8a74f9b4236ca7bcc72dd207fca23c2bf6a7c801
Co-Authored-By: Tomáš Janoušek <tomas.janousek@gooddata.com>
Inject the template-name into parameters earlier, so it's possible to
use it in the global defaults. Mark the template name as 'verbatim' so
the interpolation is skipped by the formatter.
Change-Id: I5d98e32ed2bedaa6b536e61fbdec589f358f861f
This template included using !include-jinja2:
"""
{{ my_var }}
"""
is rendered the same as an existing template that looks like this:
"""
{my_var}
"""
This also allows the use of Jinja2's richer syntax:
"""
{% for test_environment in configuration.get("envs", ["py35"]) %}
tox -e {{ test_environment }}
{% endfor %}
"""
Story: 2001135
Change-Id: Ia3ee21822d6e9237f5ea46796bc8810ecac61e2c
Problems occurring deep in the code frequently need to be caught and
additional information included in the error output to help debug
issues.
Change-Id: I5aee523a3cf9e1cb7573fa6fc5a06dc3888be1ef
Provide syntax support for specifying default values to be substituted
for variables during deep_format when no other replacement is provided.
Allows for individual variables to have a default or be optionally blank
should nothing be placed after the custom specifier.
Change-Id: Ib97a33a2bbca123791d4ca6ef5248ed200992565
To allow filenames referenced by the application specific yaml tags to
contain template job variables, use a lazy loading object that provides
a format method that can be called by the deep_format function.
Instead of processing the file, when a KeyError occurs on attempting to
call format on the filename after the yaml tag, create a LazyLoader
instance to wrap the data and provide a format method that can be called
at a later stage.
In order to call the correct method on the original Loader class,
LazyLoader needs to be given the custom tag class, a reference to the
loader and the node object. Using the tag class it can call the
from_yaml() method with the loader and node object to return the file
contents.
Since the result from the LazyLoader instance is triggered by calling
the format method, there is no need to escape the brackets used by
pythons format method since the output will not be passed through it.
In order to ensure this behaviour, nodes passed to the method handling
the '!include-raw-escape:' tag class, which need to use the LazyLoader
approach will convert to the '!include-raw:' tag class to the
LazyLoader initialization instead.
Due to a bug in sphinx with use of 'note' admonitions and manpage
generation, need to update to a version >= 1.2.1.
Change-Id: I187eb83ba54740c2c1b627bc99c2d9769687fbc7
Story: 2000522
In Python 3, BaseException.message is removed.
So we use args attribute instead.
Change-Id: Ief654970cb29e88c21add1c3d4235b86ddc0b94e
Related-Change-Id: I8f97c0b0452f348acef20e9d248f02b83cec2562
Ensure that we use the original object to create the new container to
place the formatted contents into thus ensuring that we treat the
objects passed in a pure duck-typing manner rather than assuming they
are basic python data types.
Change-Id: Ib0ccea728f08ff4cf43b8ccc92477b7c46296ce9
The goal of this patch is simply to move some classes out of
jenkins_jobs.builder into more appropriately-named modules. This started with
simply moving YamlParser into jenkins_jobs.parser but led to other moves in
order to avoid cyclic imports since YamlParser uses other classes previously
defined in jenkins_jobs.builder.
That said, this patch doesn't intend to address all of the clutter in
jenkins_jobs.builder, mostly just what is necessary to get started working on
YamlParser independent of other classes in that module.
Change-Id: Ie88bf683e495033eb0b670fe29c256a70282735f
Vsevolod Fedorov