Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
Sphinx was pinned to >1.5.0 <=1.7.0 4 years ago. It is now on v4.4.0,
and we're seeing an incompatibility with the latest version of jinja2.
Along with updating the Sphinx version, this also includes fixes for
several warnings generated in the latest version:
* Outdated docs due to the removal of the "parser" param from
Base.get_xml
* Class lookup issues
* External links for which we have extlinks defined
* Deprated PyModulelevel is replaced by PyFunction
* sphinx.ext.autodoc.Documenter.get_doc signature changed
Signed-off-by: Eric Ball <eball@linuxfoundation.org>
Change-Id: Iba65fb724f098f3e4a0b20f2a183cb2ea3c450ea
Removed "add_nonblank_xml_subelement" from the base module and
updated EnvInject to make use of "convert_mapping_to_xml" by
configuring the parameter "fail_required=False" and
setting the default options field to "None".
Change-Id: Idbe7f8b5ce242fa24adeeebbaacfe53cc09d8867
Create the ModuleRegistry anywhere other than inside the YamlParser
class. This will make it slightly easier to factor a XmlGenerator out
of YamlParser, but I also want to work toward eliminating the circular
references between YamlParser and ModuleRegistry which have been
making it difficult to understand overall program flow.
This commit also replaces all YamlParser instances being passed to
Jenkins job config generating functions with a ModuleRegistry. Mostly
it seems like the parser was only needed to call the ModuleRegistry's
'dispatch' method which to be honest I don't fully understand. This is
where the circular references mentioned in previously come in...it
seems like the "dispatch" function needs access to the (mostly) raw
data contained by the parser, so it took that as a parameter.
The need for the YamlParser's job data can be satisfied by assigning
it to a property on the ModuleRegistry object before Yaml expansion or
XML generation begins; by doing this, we allow the ModuleRegistry to
avoid referencing the parser.
Change-Id: I4b571299b81e708540392ad963163fe092acf1d9
Some Jenkins plugins depend on other plugins, and their configuration
section is a mix of both plugins.
For Jenkins Job Builder, that means reusing one component directly from
another one.
Driving the generation of XML markup is the job of Base._dispatch.
Unfortunately, components do not have access to their module object,
and even if their could, _dispatch would still be a non-public method.
Refactor Base._dispatch into ModuleRegistry.dispatch, which can be used
from any place where the parser is available.
Base and ModuleRegistry are extended so that the registry can discover
which entry point must be used for each module, if appropriate.
ModuleRegistry.dispatch signature can be simplified by dropping
component_list_type parameter.
Change-Id: Ie9d090817d0c2d464745b5634a22d3cea6a47ab1
Reviewed-on: https://review.openstack.org/26051
Reviewed-by: James E. Blair <corvus@inaugust.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Clark Boylan <clark.boylan@gmail.com>
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Tested-by: Jenkins
Using the Inject functionality with the builders section caused
an error in Jenkins because the script-file was blank but the tag
existed in the xml.
Fixing this by adding a function to create the XML only if the given
value is not None and not a blank value '':
def add_nonblank_xml_subelement(parent, tag, value):
Change-Id: I9151f7444f6807e929721b04fb8e7ad3a867cd6c
Reviewed-on: https://review.openstack.org/18820
Reviewed-by: Jon-Paul Sullivan <jonpaul.sullivan@hp.com>
Reviewed-by: Tollef Fog Heen <tfheen@varnish-software.com>
Reviewed-by: Jeremy Stanley <fungi@yuggoth.org>
Approved: Monty Taylor <mordred@inaugust.com>
Reviewed-by: Monty Taylor <mordred@inaugust.com>
Tested-by: Jenkins
Move test.sh to the tools directory.
Move parameters and notifications to their own modules; even
though they are implemented as Jenkins properties, they make
more sense as separate entities in the job builder, because
that's they way they are specified in the YAML. All three
modules that touch the properties xml object know how to
create it if it's missing.
Change-Id: I4b42ff10a93fd3ed98f632b58e47f3e0e45086d6
Reviewed-on: https://review.openstack.org/12741
Reviewed-by: Clark Boylan <clark.boylan@gmail.com>
Approved: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Switch to using entry points for loading modules as well as
individual buliders, triggers, publishers, etc.
Remove most openstack-specific python code.
Change templating so it's less repetitive -- a single project
definition will suffice for multiple jobs or job-groups.
This outputs XML that is identical to the current production XML,
warts and all. There are significant improvements that can be made
to the YAML in a separate change, as they will cause minor changes
to existing jobs (adding timestamps, logrotate, etc.). These are
mostly marked with TODO in this change.
Change-Id: Idcfddb3b43b6cfef4b20919a84540706d7a0a0b1
Reviewed-on: https://review.openstack.org/11000
Approved: James E. Blair <corvus@inaugust.com>
Reviewed-by: James E. Blair <corvus@inaugust.com>
Tested-by: Jenkins
Vsevolod Fedorov