Expand variables inside macros without parameters and jobs
the same way as they are expanded inside macros with parameters
and job templates.
Make tags behave inside macros without parameters and jobs
the same way as they are expanded inside macros with parameters
and job templates.
Update or fix affected tests.
Story: 2010588
Story: 2010963
Story: 2010535
Task: 47394
Task: 49069
Task: 47151
Change-Id: Ie05ae6aa386c62ebbf68dd3e2c7001a4e444a47a
Folders was not actually tested by tests. Fix tests and add more tests.
Fix bug in code unearthed by now-working tests: defaults were not checked
for job folders.
Task: 49188
Change-Id: I922af7a28b0ec0c99ef3f8a17b3d5a9c10f2dce0
Rewrite YAML parser, YAML objects and parameters expansion logic to
enable better control over expansion logic.
Broken backward compatilibity:
* More agressive parameter expansion. This may lead to parameters
expanded in places where they were not expanded before.
* Top-level elements, which is not known to parser (such as 'job',
'view', 'project' etc), are now lead to parse failures.
Prepend them with underscore to be ignored by parser.
* Files included using '!include-raw:' elements and having formatting in
it's path ('lazy-loaded' in previous implementation) are now expanded
too.
Use '!include-raw-escape:' for them instead.
See changes in these tests for examples:
tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs-multi002.yaml
tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
* Parameters with template value using itself were substituted as is.
For example: "timer: '{timer}'" was expanded to "{timer}". Now it
leads to recursive parameter error.
See changes in this test for example:
tests/yamlparser/job_fixtures/parameter_name_reuse_default.*
->
tests/yamlparser/error_fixtures/parameter_name_reuse_default.*
* When job group includes a job which was never declared, it was just
ignored. Now it fails: job is missing.
See changes in this test for example:
tests/yamlparser/job_fixtures/job_group_includes_missing_job.*
->
tests/yamlparser/error_fixtures/job_group_includes_missing_job.*
Change-Id: Ief4e515f065a1b9e0f74fe06d7e94fa77d69f273
Rename fixtures directory to job_fixtures to prepare for adding view_fixtures
directory in the following commits.
Change-Id: Ic20997cae020b542ddc22bf444fa6b92fbcae064
The links pointing to full example and anchor & aliases on
yaml website are no longer valid, causing failures for
docs-linkcheck. This change updates the links in the doc.
curl https://yaml.org/spec/1.2/#id2761803
<meta http-equiv="refresh" content="0; url=/spec/1.2.2/" />
curl https://yaml.org/spec/1.2/#id2765878
<meta http-equiv="refresh" content="0; url=/spec/1.2.2/" />
Change-Id: I1d57e9b414694e2dd8e0dca24ef2e6ea96cb35a0
It looks like the YAML specifications are no longer available in:
yaml.org/spec/1.2/spec.html
but in:
yaml.org/spec/1.2/
See:
$ curl https://yaml.org/spec/1.2/spec.html
<meta http-equiv="refresh" content="0; url=https://yaml.org/spec/1.2/" />
Update these links to avoid errors reported by jjb-tox-docs-linkcheck.
Signed-off-by: Matthieu Baerts <matthieu.baerts@tessares.net>
Change-Id: Ie928adf2e4c321e900bfbf36c19ae34a3dc7311d
The Jenkins Wiki page is deprecated and Jenkins community is expected
to migrate their documentation to plugins.jenkins.io URL. This patch
updates all publishers that have a relevant plugins.jenkins.io URL
documentation link.
Change-Id: I94448d04f2b28fbf3befde2e49c76d196e352d7e
Signed-off-by: Thanh Ha <zxiiro@gmail.com>
Took inspiration from my StackOverflow reply here
https://stackoverflow.com/questions/50177308 to document a practical
use of setting variable defaults and inheritenence.
Change-Id: I45934341d3ff3fbf4ff711a435924197d2f12712
Signed-off-by: Thanh Ha <zxiiro@linux.com>
Allow views to also be configured via templates similar to
job-templates.
This adds a new project key called "views" and parser type called
"view-template" allowing the user to add custom views.
Example:
- view-template:
name: '{name}-template-{seq}'
description: 'testing view templates feature'
view-type: list
regex: 'test-view-.*'
- project:
name: 'test-view'
views:
- '{name}-template-{seq}'
seq:
- a
- b
- c
Change-Id: I6eb225f24bc3c7d790c7dcab6391735c579ac71a
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
View documentation was not properly being generated and linked
in the index. Let's add that in.
Change-Id: I6197dbecc89ddcf8c6f6b0e6a3d4c517224dc77b
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Inject the template-name into parameters earlier, so it's possible to
use it in the global defaults. Mark the template name as 'verbatim' so
the interpolation is skipped by the formatter.
Change-Id: I5d98e32ed2bedaa6b536e61fbdec589f358f861f
Allows specifying a folder attribute for each job generated, which in
turn is used when creating or uploading to place the job under the
requested folder.
The job name is expanded after defaults are applied, to support the
attribute being defined within a set of defaults applied to a number of
jobs.
This in turn allows for multiple jobs with the same basename to exist,
provided they are targeted at different folders.
Does not support creating the folders if they do not exist.
Change-Id: I8c2157c4c81087cc972a048d1b88d5f08ac65361
Provide syntax support for specifying default values to be substituted
for variables during deep_format when no other replacement is provided.
Allows for individual variables to have a default or be optionally blank
should nothing be placed after the custom specifier.
Change-Id: Ib97a33a2bbca123791d4ca6ef5248ed200992565
- Adds the ability for JJB to work with views
- Views can be created, updated, and deleted.
- New modules for List view and Build Pipeline view are added
- New tests for testing the deletion of views
Example View configuration:
- view:
name: MyView
view-type: list
Change-Id: Idb29a4407bcc14593e10a4d951036cb04e8e6c27
Co-Authored-By: Brandon Leonard <brandon.leonard@rackspace.com>
Co-Authored-By: Joao Vale <jpvale@gmail.com>
Co-Authored-By: Lucas Dutra Nunes <ldnunes@ossystems.com.br>
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>
Document the ability to define default values for template variables
within the template definition thus removing the need to always having
to set such values in job definitions.
Change-Id: If4d8bfd898066b8e6350ed22f0c97eb939ae4348
It is useful to allow templates that can have different settings for
slightly different jobs while still following the same naming scheme.
Additionally this allows for shorter and more descriptive names to be
used without the confusion of the variable placeholders, and for the
resulting job names from the templates to be altered without needing to
update all references.
Change-Id: Idc3517b44873210a33f988ebff449ea2ed567054
This adds some notes on parameter expansion with macros and especially
calls out some of the details around using "{" with shell-scripts.
Change-Id: Ibef92a60b9f44f7dc70db65aff7d97b3a665a6bf
Add documentation to let users know that they can also define jobs
in JSON. Create a new test suite to verify that jjb can correctly
generate jenkins xml from json definitions.
Change-Id: Idb5216b0b56837d0f4dd2298986149be66bd217e
Anchors and aliases were expanded internally within JJB's yaml loading
calls so they were limited to individual documents. Now, included files
will have access to aliases of anchors already defined at previously
processed files.
Example:
- default:
name: default-timeout-wrapper
timeout: &timeout
fail: true
elastic-percentage: 150
elastic-default-timeout: 90
type: elastic
- wrapper: !include include002_1.yaml.inc
Previously was not possible to use '*timeout' alias inside
include002_1.yaml.inc file
Closes-Story: 2000173
Change-Id: Ic031ddbb0310bd11748183fbde9502735c3b7169
'raw' allow users to have a fallback in case
a plugin is not yet supported or the plugin is not generating the
expected result.
Only intended as a last fallback, but useful when waiting for
review to complete.
Change-Id: If0d22d7d43d35649e78aa7481e1c0f1ed21a6025
Added a new configuration option under the section "job_builder" named
"allow_empty_variables" that if set to true, will replace non existing variables
in strings with the empty string instead of raising an error.
It's very useful if you have a shell script, that has optional values, for
example:
EXTRA_PACKAGES=({extra-packages})
for package in "${EXTRA_PACKAGES[@]}"; do
install "$package"
done
Or modifying the script behavior with flags, with empty as default value:
WITH_AUTOGEN={with-autogen}
if [[ $WITH_AUTOGEN ]]; then
./autogen.sh
fi
Then if you have two different jobs that use that script in their builder, you
just have to set the extra parameter or ignore it to change the behavior:
- builder:
name: mybuilder
builders:
- shell: !include shell-scripts/myscript.sh
- job-template:
name: 'mytpl-{name}'
...
builders:
- mybuilder
- project:
name: myproj1
with-autogen: true
jobs:
- 'mytpl-{name}'
- project:
name: myproj2
extra-packages: |
extrapkg1
extrapkg2
Change-Id: Iad9f0e522725e6fd6681cd62d3e36f69baf09585
Signed-off-by: David Caro <dcaroest@redhat.com>
Adds two tests to ensure correct behaviour with referencing yaml
anchors and a third test to verify the expansion internally of yaml
anchors and aliases.
* test that anchors are not carried across subsequent top level
invocations of yaml.load(). This will be used subsequently to ensure
that where anchors are allowed across included files they are
correctly reset on each top level call.
* test that where anchors are defined in a top level file and
subsequently included files, duplicate anchors raise exceptions as
though they were defined within the same file.
* test that data returned from yaml loading contains the additional data
specified by the alias. Uses json to force conversion so that the
outputted yaml contains the results of the anchors and aliases instead
of them.
Update documentation to contain details of the use of anchors and
aliases including a refernce to a simple generic example from the
specification as well as a JJB specific example.
Change-Id: I0f2b55e1e2f2bad09f65b1b981baa0159372ee10
This patch enables parameters to be expanded inside of other parameters. For
instance:
- job-template:
name: '{value-stream}_{pipeline_type}_foo'
builders:
- shell: 'echo "I am foo job!"'
publishers:
- trigger-parameterized-builds:
- project: '{downstream}'
current-parameters: False
condition: ALWAYS
git-revision: True
- job-template:
name: '{value-stream}_{pipeline_type}_bar'
builders:
- shell: 'echo "I am bar job!"'
publishers:
- trigger-parameterized-builds:
- project: '{downstream}'
current-parameters: False
condition: ALWAYS
git-revision: True
- job-template:
name: '{value-stream}_{pipeline_type}_baz'
builders:
- shell: 'echo "I am baz job!"'
publishers:
- trigger-parameterized-builds:
- project: '{downstream}'
current-parameters: False
condition: ALWAYS
git-revision: True
- job-group:
name: 'pipeline2'
pipeline_type: 'p2'
jobs:
- '{value-stream}_{pipeline_type}_foo':
downstream: '{value-stream}_{pipeline_type}_bar'
- '{value-stream}_{pipeline_type}_bar':
downstream: '{value-stream}_{pipeline_type}_baz'
- '{value-stream}_{pipeline_type}_baz':
- job-group:
name: 'pipeline1'
pipeline_type: 'p1'
jobs:
- '{value-stream}_{pipeline_type}_baz':
downstream: '{value-stream}_{pipeline_type}_bar'
- '{value-stream}_{pipeline_type}_bar':
downstream: '{value-stream}_{pipeline_type}_foo'
- '{value-stream}_{pipeline_type}_foo':
- project:
name: derp
jobs:
- 'pipeline1':
value-stream:
- experimental
- staging
- 'pipeline2':
value-stream:
- experimental
- staging
- production
Prior to this commit, the setup above was not possible because the `downstream`
defined as parameters to be passed to job templates in the job groups would not
have the `value-stream` and `pipeline-type` parameters interpolated. The
trigger-parameterized-jobs `project` value for each of the 15 jobs created with
this setup would all look something like `{value-stream}_{pipeline_type}_foo`
which does not actually point to any particular job.
The only way I was previously able to specify a downstream job with parameters
was to put the `{value-stream}_{pipeline_type}_foo` string directly in the
job-template which meant that job templates were statically bound to each
other--I had no reasonable way to re-order or easily select subsets of the jobs
at the job-group or project level as can be seen here in the difference between
`pipeline1` and `pipeline2`.
Change-Id: I26dfb112cd4842c40c3d561f8de89f7654a07b8f
This change adds a test to validate the general job parameters and uses
that test as documentation for the general module. It also replaces the
doc of the general params in the top level definitions.rst with references
to the general module.
Change-Id: I3df04d69186ca49d7297f8141855a4c72c73be1e
I noticed that users were not really aware of how this feature
worked so added additional documentation for it.
Change-Id: Ic5385ae859295cdd015d698ede3da4988efd9ca7
Closes-Bug: 1335945
Vsevolod Fedorov