Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
Pytest makes each scenario into individual selectable test.
To be able to run each scenario separately is very useful for development.
Change-Id: I4b1c990a1fd839ce327cd7faa27159a9b9632fed
Before this change using an anchor/alias in !j2-yaml resulted in an
exception, even if retain_anchors was set to True:
> File "(...)/jenkins-job-builder/.tox/py38/lib/python3.8/site-packages/yaml/composer.py", line 68, in compose_node
> raise ComposerError(None, None, "found undefined alias %r"
> yaml.composer.ComposerError: found undefined alias '<alias-name-xxx>'
The reason was that LateYamlLoader.get_object_to_format() was calling
load() without retain_anchors=True so effectively it was resetting
anchors during deep formatting...
From now on yaml.load() is called directly to avoid unwanted
side-effects of load() which is meant to be called from other modules,
typically by parser.YamlParser().
Change-Id: I0fde43c5c27ddf78f18dc244d8dba5bd768306b7
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.
Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Preparatory move for adopting pre-commit hooks.
Shoudl only remove newlines and spaces at end of files.
Change-Id: Ia33679467c533722b9308f7e7b74be395c926428
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
If set to True, YAML anchors can be referenced across files, allowing jobs to be
composed from bits of YAML defined in separate files. False by default.
Story: 2000338
Task: 2547
Change-Id: I034ce3bce0030093cb8d4266dabbdb06d96306d6
Adds a yaml application tag '!join:' to support defining data as a list
while having the resulting contents automatically joined together with
the specified delimiter and passed as a string into the resulting object
returned by yaml for JJB to process.
This allows users to store long lists of data that is combined together
for use with Jenkins plugins that expect a delimited string of arguments
instead of needing each module in JJB to provide individual support.
Change-Id: I745181ade3926d5c29708963189ae499a0378ece
Refactor base test class inheritance to allow for BaseTest class to be
reused without needing to also import the TestCase class everywhere in
order to avoid having the common test function executed on base classes.
This makes it easier to build base testing classes and then simplifies
the subclassing of these for actual tests.
Change-Id: I89809e8082469f814f245db4a9ab7658aac8a405
This is the first step in removing jenkins_modules.parser.YamlParser
references from jenkins_jobs.builder.Builder entirely, necessary
because load_files is actually a YamlParser behavior, not a Builder
behavior.
Change-Id: I1cad99b4cdb44af25ba398837f7f328cfcbd5bbb
This commit sees JJBConfig start to take the form it ought to have,
namely using attributes to represent different logical sections of
configuration that target specific subsystems of JJB.
It also moves ConfigParser data retrieval from
jenkins_jobs.modules.helpers.get_value_from_yaml_or_config_file() to
jenkins_jobs.config.JJBConfig.get_module_config()
TODO: Add JJBConfig tests to validate the behavior of JJBConfig in
specific circumstances.
Change-Id: I053d165559f5325a2f40b239117a86e6d0f3ef37
Capture logs to attach them on exceptions to simplify the debugging of
errors in tests when they occur.
Change-Id: I5c1b5c306aba37789a8d41eabf65e0ac628af838
Implement custom tags for including files by subclassing YAMLObject to
allow PyYAML to easily convert from the Yaml nodes.
Change to support both lists and strings under the same tag name and
deprecate the old tag naming. Rename tests using the deprecated tags to
ensure they are not used for examples. Remove old tests that simply
duplicate others.
Change-Id: I1a8d3376ea6e4918a2adb05fb856b939a3124d74
Remove the test options suppressing ignoring differences in whitespace
outputted in the XML to allow capturing of issues where the code
incorrectly adds or removes significant whitespace.
Add a helper function to deal with the edge case of parsing XML directly
that contains non significant whitespace to avoid accidentally adding
excess whitespace when minidom outputs the resulting document.
Change-Id: I9936042cd82c204ba2b3c19f575703e33564f7fd
Anchors and aliases were expanded internally within JJB's yaml loading
calls so they were limited to individual documents. Now, included files
will have access to aliases of anchors already defined at previously
processed files.
Example:
- default:
name: default-timeout-wrapper
timeout: &timeout
fail: true
elastic-percentage: 150
elastic-default-timeout: 90
type: elastic
- wrapper: !include include002_1.yaml.inc
Previously was not possible to use '*timeout' alias inside
include002_1.yaml.inc file
Closes-Story: 2000173
Change-Id: Ic031ddbb0310bd11748183fbde9502735c3b7169
Add variants of the include-raw* tags, which accept a list of files.
Those files are concatenated and included as a string data into the
calling yaml construct.
Change-Id: I6af87a298268acc8a73d7a2b50f9f99733d8723a
Replace requiring empty files for tests that produce no output and treat
any non-existant output file candidate as though the output is blank.
This simplifies tests that are intended to trigger exceptions and will
not generate an output.
Change-Id: I978fdabb488bc3e4bc6f22db294d21ca5d68913b
Adds two tests to ensure correct behaviour with referencing yaml
anchors and a third test to verify the expansion internally of yaml
anchors and aliases.
* test that anchors are not carried across subsequent top level
invocations of yaml.load(). This will be used subsequently to ensure
that where anchors are allowed across included files they are
correctly reset on each top level call.
* test that where anchors are defined in a top level file and
subsequently included files, duplicate anchors raise exceptions as
though they were defined within the same file.
* test that data returned from yaml loading contains the additional data
specified by the alias. Uses json to force conversion so that the
outputted yaml contains the results of the anchors and aliases instead
of them.
Update documentation to contain details of the use of anchors and
aliases including a refernce to a simple generic example from the
specification as well as a JJB specific example.
Change-Id: I0f2b55e1e2f2bad09f65b1b981baa0159372ee10
Python 3 enables hash randomization by default, additionally tox 1.7
turns on the same randomization for earlier versions of python by
default. Need to ensure that order of iteration over the yaml data and
resulting XML has deterministic order for testing.
Adapts https://gist.github.com/enaeseth/844388 which ensures data read
by yaml will have its order retained in a predictable manner across
multiple python versions.
Additionally it seems more sensible to ensure that the order of
generated XML snippets corresponding to the input yaml files are
consistently in the same order as the entries in the source files.
Closes-Bug: #1333349
Change-Id: I6bf6d298a2609cc6ddbbc6b02b7f1a04413a5c89
Add support for local tags which are application specific to allow
including of other yaml files or code from scripts. Allows for code to
be maintained and tested as seperate files, as well as reduces
duplication of yaml code that cannot be macro'ed or easily templated by
including it from a common file.
Adds support for the following tags:
'include' - load file as yaml code
'include-raw' - load file as data (for scripts)
'include-raw-escaped' - load file as data with escaping braces '{}'
as default for use with job-templates
Use configuration file options to provide a search path for the files.
- Test behaviour of yaml tags independent of any XML generation
by comparing json result of yaml parsing to verify that certain
tags do/don't recall the yaml.load() method.
- Add examples for the include tags via addition tests for YamlParser
class
Inspired by
http://stackoverflow.com/questions/528281/how-can-i-include-an-yaml-file-inside-another
Change-Id: Ib90a07043112d4739d6529ceddbc9817668bcec0
Vsevolod Fedorov