Expand variables inside macros without parameters and jobs
the same way as they are expanded inside macros with parameters
and job templates.
Make tags behave inside macros without parameters and jobs
the same way as they are expanded inside macros with parameters
and job templates.
Update or fix affected tests.
Story: 2010588
Story: 2010963
Story: 2010535
Task: 47394
Task: 49069
Task: 47151
Change-Id: Ie05ae6aa386c62ebbf68dd3e2c7001a4e444a47a
LegacyVersion class is removed from newer setuptools package. But
support for legacy versions is added in python-jenkins 1.8.2.
Switch to that implementation.
Fix broken plugin version comparison for legacy versions.
Assume latest plugin version if no plugin version is found.
Story: 2010990
Story: 2009943
Story: 2009819
Story: 2010842
Task: 49236
Task: 44852
Task: 44396
Task: 48448
Change-Id: Id7f0be1c42357454bd9bedcdee3fefb174943d81
Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
Plugins are no longer following semantic versioning, and frequently
include additional versioning information.
pkg_resources.parse_version generally handles this well, but some
versions with a hyphen will return a LegacyVersion object rather than
a Version object, causing it to always compare as lower than any
Version object.
Simply replacing any '-' with '+' fixes this issue, and does not
materially change the versioning.
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Change-Id: I5ad949d688ce7ebd0d183d69f4ce87b35343357f
Sphinx was pinned to >1.5.0 <=1.7.0 4 years ago. It is now on v4.4.0,
and we're seeing an incompatibility with the latest version of jinja2.
Along with updating the Sphinx version, this also includes fixes for
several warnings generated in the latest version:
* Outdated docs due to the removal of the "parser" param from
Base.get_xml
* Class lookup issues
* External links for which we have extlinks defined
* Deprated PyModulelevel is replaced by PyFunction
* sphinx.ext.autodoc.Documenter.get_doc signature changed
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Change-Id: Iba65fb724f098f3e4a0b20f2a183cb2ea3c450ea
Due to the fact how dispatching works the 'authorization' property
handler was not always invoked through Properties.gen_xml(), leading to
a bug and an invalid test case: project-with-auth-properties.(yaml/xml)
This change pushes the logic for determining if the object is a
folder/multi-branch project from Properties.gen_xml() to the
authorization() function itself. For that to work the authorization()
function needed access to the top-level job object, which is now
conditionally passed to each dispatched function as a keyword argument,
if the function takes 'job_data' argument. Note that taking this
argument is completely optional so no changes were required in other
handlers. In the future the same approach could be taken to eliminate
the hacks for 'uno-choice' in Parameters.gen_xml().
Additionally ModuleRegistry.dispatch() now merges the top-level job
object with any template data before deep-formatting, so that job-level
properties are now available in Jinja templates. A very nice use case
is in project-with-auth-j2-yaml.yaml test case.
Change-Id: I9a49de74055cd9acfdc87dbad1fc454548643e8f
Resolves flake8 BLK related issues.
"BLK100 Black would make changes."
"BLK999 Unexpected exception: ..."
Change-Id: Ic6d4265e3d02b2f2a15ab851eaed98288c62730c
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Each component (builder, publishers...) is loaded everytime dispatch()
is called. Its main purpose is to lookup the type of the component.
However pkg_resources.EntryPoint.load() is a costly operation and it was
done every single time dispatch() is called.
Against Wikimedia configurations files [0] it is extremely noticeable.
The entrypoint is loaded solely to figure out its type. I do not see any
good reason for it to change, thus introduce a new cache
_component_type_cache.
That then let us prevent invoking load() every single time.
Run time to generate Wikimedia configuration files went from 17 seconds
to a mere 4 seconds!
[0] https://gerrit.wikimedia.org/g/integration/config/+/master/jjb/
Change-Id: I49bc5dc91e3ba69b7f15c5f5ded8a42fb61a60c2
When profiling a JJB run against Wikimedia configuration files [0], I
noticed a lot of calls to package_resource trying to figure out the
version.
Years ago, I already added coded to cache the entrypoints, unfortunately
only the entry point definition got cached not the underlying function.
Thus each entry point ends up invoking a load() which dramatically slow
down JJB.
A low hanging fruit is to invoke load() and cache the returned function.
Doing so, the run time goes down from 28 seconds down to 17 seconds.
[0] https://gerrit.wikimedia.org/g/integration/config/+/master/jjb/
Change-Id: I336b556f28328d8acbdd71c43c8775d55c3720d8
Please review the following patch containing the code changes in
the repo. This patch is a transition patch and is the auto-generated
output of the python-black tool.
Change-Id: I2d2de71da8a105fb62b561899ae78441ddab4032
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
As Jinja2 provides functionality other than interpolation, it's entirely
feasible to use it outside of a context where there are template
variables. This change ensures that we always deep format Jinja2
templates, even when we wouldn't normally interpolate a regular JJB
template.
Change-Id: Ia3f86dd2073d48594eaf791fa2277db7e5ebf0ab
Introduce the registry.MacroRegistry class to handle:
* registration of macro types via setuptools' entrypoints
* registration of individual macros for lookup by component list type
* expansion of macros references during YAML "parsing"
As a consequence there is a reduction in performance due to moving the
expansion of macros from inline with XML generation, to requiring
multiple passes over macro component lists.
This decrease in efficiency results in approx ~30-50% increase in unit
test time. Since this will allow for jobs to be expanded from
templates/macros in parallel with future changes, it is a reasonable
short term trade-off as the most computationally expensive task is
updating the definitions on the remote master
Change-Id: I292c6b1f8472370282205426cd8ceb847eb969bd
when building snapshots, maven will sometimes add some information
*after* the -SNAPSHOT like this:
1.4.6-SNAPSHOT (private-0986edd9-egolov)
the old registry code would transform this to
1.4.6.preview (private-0986edd9-egolov)
which is not a valid version string for pkg_resources.
strip everything after the -SNAPSHOT tag and thus produce a clean
version string that is comparable properly
Change-Id: I765c991ac7632b3c76f548112d2a5323a8ce4489
Signed-off-by: Evgeni Golov <evgeni@golov.de>
Problems occurring deep in the code frequently need to be caught and
additional information included in the error output to help debug
issues.
Change-Id: I5aee523a3cf9e1cb7573fa6fc5a06dc3888be1ef
Create the ModuleRegistry anywhere other than inside the YamlParser
class. This will make it slightly easier to factor a XmlGenerator out
of YamlParser, but I also want to work toward eliminating the circular
references between YamlParser and ModuleRegistry which have been
making it difficult to understand overall program flow.
This commit also replaces all YamlParser instances being passed to
Jenkins job config generating functions with a ModuleRegistry. Mostly
it seems like the parser was only needed to call the ModuleRegistry's
'dispatch' method which to be honest I don't fully understand. This is
where the circular references mentioned in previously come in...it
seems like the "dispatch" function needs access to the (mostly) raw
data contained by the parser, so it took that as a parameter.
The need for the YamlParser's job data can be satisfied by assigning
it to a property on the ModuleRegistry object before Yaml expansion or
XML generation begins; by doing this, we allow the ModuleRegistry to
avoid referencing the parser.
Change-Id: I4b571299b81e708540392ad963163fe092acf1d9
Remove reference to ConfigParser object in ModuleRegistry. Instead:
* make use of JJBConfig.get_module_config to grab settings for Hipchat
Notifier Plugin
* make use of JJBConfig.yamlparser['allow_empty_variables'] rather
than repeating ConfigParser logic moved out of the YamlParser into
JJBConfig in an earlier commit.
Change-Id: Icb7ef514826005545e48af993335ce120f973b0d
Currently JJB warns over 700 times when checking project-config with:
WARNING:jenkins_jobs.registry:You have a macro ('tox') defined for
'builder' component type that is masking an inbuilt definition
Warn only once for this.
Change-Id: I010765269c499320fc9a84b9493d22949080e9a0
Prefer user defined macros over inbuilt entry points, as JJB may
accidentally use a name that has already been used by an end-user to
define a custom macro.
Warn users when they have a macro defined that masks an in-built
component to avoid accidentally changing XML generated with new
releases.
Change-Id: I6cae62d7cc40be6c4a5636a74f151fcce4cdd856
Use the component type entry points to infer modules to scan
corresponding modules and automatically construct entry points for any
functions associated with yaml entries in JJB definitions.
This removes the need for new JJB modules added directly to the project
to also need to be listed in the setup.cfg to be reachable. Simply
adding the needed functionality to the correct file ensures that it is
available.
Change-Id: I2a0fd8f6720ddfd8582b58e47556a87ed1ad553d
Prior to this patch macros serialized into a yaml document, a simple string
substitution was applied and then they were re-loaded into python. This
created several problems:
- newlines are lost because the yaml loading was not respecting the
newline settings of the original strings (Story: 1387060)
- {obj:} notation was not working for macro parameters (Story: 2000254)
- multiline strings passed into scripts will create a yaml syntax error
when being re-loaded (unticketed)
This patch resolves these problems by replacing the previous interpolation
code with the more robust `deep_format` method. A side effect of this patch
is that there are now fewer disparate code paths for interpolating variables.
Change-Id: Ia06c98286537a50dc52c6e5e000b32ebc9e8ede2
Signed-off-by: Justin Stoller <justin.stoller@gmail.com>
The goal of this patch is simply to move some classes out of
jenkins_jobs.builder into more appropriately-named modules. This started with
simply moving YamlParser into jenkins_jobs.parser but led to other moves in
order to avoid cyclic imports since YamlParser uses other classes previously
defined in jenkins_jobs.builder.
That said, this patch doesn't intend to address all of the clutter in
jenkins_jobs.builder, mostly just what is necessary to get started working on
YamlParser independent of other classes in that module.
Change-Id: Ie88bf683e495033eb0b670fe29c256a70282735f
Vsevolod Fedorov