Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.
Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
This also adds support for using custom loaders in job definitions
not just job-templates and builders. I have also added unit tests for include-jinja2-as-yaml custom loader.
Example usage:
- axis:
type: user-defined
name: VERSIONS
values:
!include-jinja2-as-yaml: versions.j2.yaml.inc
Where versions.j2.yaml.inc is
{% for possible_versions in configuration["possible_versions"] %}
- {{ possible_versions["versions"] }}
{% endfor -%}
Change-Id: I15a431d5a86b29d704efda8576965ade2b7dcd2f
Similar to job updates when passing a specific job name in the CLI,
views should also be ignored if they were not explicitly specified.
Change-Id: I77488b0af453964f77bc8d565d43f404327ef139
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
The random print statement does not provide any useful information
and spams the console output if a repo has many declared views with
uninteresting output.
Change-Id: I7f40730c753ec8f1145ba6144fe28bf97844b9e2
Signed-off-by: Thanh Ha <zxiiro@linux.com>
- upgrades hacking to current version
- sorts new linting issues
- sorts bug with sys.reload on py3
Change-Id: I4a18abc93116667a2733e8aec619ac59ea73d630
Signed-off-by: Sorin Sbarnea <ssbarnea@redhat.com>
If set to True, YAML anchors can be referenced across files, allowing jobs to be
composed from bits of YAML defined in separate files. False by default.
Story: 2000338
Task: 2547
Change-Id: I034ce3bce0030093cb8d4266dabbdb06d96306d6
Allow views to also be configured via templates similar to
job-templates.
This adds a new project key called "views" and parser type called
"view-template" allowing the user to add custom views.
Example:
- view-template:
name: '{name}-template-{seq}'
description: 'testing view templates feature'
view-type: list
regex: 'test-view-.*'
- project:
name: 'test-view'
views:
- '{name}-template-{seq}'
seq:
- a
- b
- c
Change-Id: I6eb225f24bc3c7d790c7dcab6391735c579ac71a
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Make use of an improved regex pattern matching to simply escape the
template name when passing into the params to allow the name be
preserved and used as input to other params.
This avoids needing to perform an additional branch test against all
variables being formatted.
Change-Id: I9c0ac8996d520b9acff3d29039c5c3d76f56d899
Depends-On: I8a74f9b4236ca7bcc72dd207fca23c2bf6a7c801
Inject the template-name into parameters earlier, so it's possible to
use it in the global defaults. Mark the template name as 'verbatim' so
the interpolation is skipped by the formatter.
Change-Id: I5d98e32ed2bedaa6b536e61fbdec589f358f861f
Allows specifying a folder attribute for each job generated, which in
turn is used when creating or uploading to place the job under the
requested folder.
The job name is expanded after defaults are applied, to support the
attribute being defined within a set of defaults applied to a number of
jobs.
This in turn allows for multiple jobs with the same basename to exist,
provided they are targeted at different folders.
Does not support creating the folders if they do not exist.
Change-Id: I8c2157c4c81087cc972a048d1b88d5f08ac65361
Introduce the registry.MacroRegistry class to handle:
* registration of macro types via setuptools' entrypoints
* registration of individual macros for lookup by component list type
* expansion of macros references during YAML "parsing"
As a consequence there is a reduction in performance due to moving the
expansion of macros from inline with XML generation, to requiring
multiple passes over macro component lists.
This decrease in efficiency results in approx ~30-50% increase in unit
test time. Since this will allow for jobs to be expanded from
templates/macros in parallel with future changes, it is a reasonable
short term trade-off as the most computationally expensive task is
updating the definitions on the remote master
Change-Id: I292c6b1f8472370282205426cd8ceb847eb969bd
Problems occurring deep in the code frequently need to be caught and
additional information included in the error output to help debug
issues.
Change-Id: I5aee523a3cf9e1cb7573fa6fc5a06dc3888be1ef
- Adds the ability for JJB to work with views
- Views can be created, updated, and deleted.
- New modules for List view and Build Pipeline view are added
- New tests for testing the deletion of views
Example View configuration:
- view:
name: MyView
view-type: list
Change-Id: Idb29a4407bcc14593e10a4d951036cb04e8e6c27
Co-Authored-By: Brandon Leonard <brandon.leonard@rackspace.com>
Co-Authored-By: Joao Vale <jpvale@gmail.com>
Co-Authored-By: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Output the variable inputs used that trigger an error when expanding
template names to the error logger channel in a sensible format.
Ensures that when indented variable inputs for templates result in
exceptions when expanding a template name, that the project, template
name and variables that failed to be iterated over are outputted in a
log error message along with the original set of inputs from the
project definition to make it easier for end users to find where the
error has been made in a JJB definition.
Add code to allow dumping of variables stored in OrderedDict
transparently to match the input format used in JJB definitions and
hide the implementation detail of using OrderedDict to be within the
localyaml library.
Change-Id: I660bb0ca3b109e1a861948d6a867f185047b90ae
Separate XML generation from Yaml parsing/interpreting. The goal here
is to allow different sources to provide data for XML generation,
including external API users writing job definitions in pure Python or
JJB developers who would like to work on alternative Yaml parsing code
since the current YamlParser has very likely reached the limits of
what can be reasonably done with one giant expandYaml procedure.
Change-Id: I9da848acac7e944c0e07286b7399b2e1956a58a5
Create the ModuleRegistry anywhere other than inside the YamlParser
class. This will make it slightly easier to factor a XmlGenerator out
of YamlParser, but I also want to work toward eliminating the circular
references between YamlParser and ModuleRegistry which have been
making it difficult to understand overall program flow.
This commit also replaces all YamlParser instances being passed to
Jenkins job config generating functions with a ModuleRegistry. Mostly
it seems like the parser was only needed to call the ModuleRegistry's
'dispatch' method which to be honest I don't fully understand. This is
where the circular references mentioned in previously come in...it
seems like the "dispatch" function needs access to the (mostly) raw
data contained by the parser, so it took that as a parameter.
The need for the YamlParser's job data can be satisfied by assigning
it to a property on the ModuleRegistry object before Yaml expansion or
XML generation begins; by doing this, we allow the ModuleRegistry to
avoid referencing the parser.
Change-Id: I4b571299b81e708540392ad963163fe092acf1d9
This is the first step in removing jenkins_modules.parser.YamlParser
references from jenkins_jobs.builder.Builder entirely, necessary
because load_files is actually a YamlParser behavior, not a Builder
behavior.
Change-Id: I1cad99b4cdb44af25ba398837f7f328cfcbd5bbb
Remove reference to ConfigParser object in ModuleRegistry. Instead:
* make use of JJBConfig.get_module_config to grab settings for Hipchat
Notifier Plugin
* make use of JJBConfig.yamlparser['allow_empty_variables'] rather
than repeating ConfigParser logic moved out of the YamlParser into
JJBConfig in an earlier commit.
Change-Id: Icb7ef514826005545e48af993335ce120f973b0d
This commit sees JJBConfig start to take the form it ought to have,
namely using attributes to represent different logical sections of
configuration that target specific subsystems of JJB.
It also moves ConfigParser data retrieval from
jenkins_jobs.modules.helpers.get_value_from_yaml_or_config_file() to
jenkins_jobs.config.JJBConfig.get_module_config()
TODO: Add JJBConfig tests to validate the behavior of JJBConfig in
specific circumstances.
Change-Id: I053d165559f5325a2f40b239117a86e6d0f3ef37
It is useful to allow templates that can have different settings for
slightly different jobs while still following the same naming scheme.
Additionally this allows for shorter and more descriptive names to be
used without the confusion of the variable placeholders, and for the
resulting job names from the templates to be altered without needing to
update all references.
Change-Id: Idc3517b44873210a33f988ebff449ea2ed567054
In complex configurations it may be useful to have a template
print out its name so that a given job can be easily traced to
its definition.
Change-Id: I1cfed9c27a58b45ba48aeed04839bfc8f54a831e
Where n = number of variables in scope for a particular job template instance.
and m = number of job template instance.
Let's imagine for a large set of jobs with n = 100 and m = 50
This debug message makes it difficult to scan through JJB output looking for
signficant debug messages and does not provide a clear value.
It is trivially easy to know which variables are and which are not included in a
job-template's name by simply reading it.
Change-Id: I745e26ac6062277fc477cb1ca70cf2809f5806e3
Ensure file objects including stdin/stdout objects are wrapped using
codecs to handle unicode translation to the selected encoding for
input/output.
Add tests to simulate different encodings for input/output and
consequently fix the reading of input from stdin. Include test to
trigger failure to encode a unicode valid character using 'ascii'
encoding.
Change-Id: I9a5c4c96d131873db0000377729b8b0a45aa470d
Use io.open() to allow reading and writing of files in 'utf-8' format
irrespective of the terminal encoding selected.
Change-Id: Ie952617a34c0719efc59a7729d698beafaa477b0
waynr fixed a bug but explained in concat_defaults002
This bug happens when some values are specified in an other defaults
namespace then global.
This causes JJB to fail that its unable to find the required keys
becasue these were not merged correctly
Change-Id: Id0df24ba7cf8281389c2ed7af7ee716047d0e4a5
The goal of this patch is simply to move some classes out of
jenkins_jobs.builder into more appropriately-named modules. This started with
simply moving YamlParser into jenkins_jobs.parser but led to other moves in
order to avoid cyclic imports since YamlParser uses other classes previously
defined in jenkins_jobs.builder.
That said, this patch doesn't intend to address all of the clutter in
jenkins_jobs.builder, mostly just what is necessary to get started working on
YamlParser independent of other classes in that module.
Change-Id: Ie88bf683e495033eb0b670fe29c256a70282735f
Vsevolod Fedorov