Commit Graph

37 Commits

Author SHA1 Message Date
Ian Wienand fc8e68f0e9
Update to ansible-lint 6.8.2
This updates to ansible-lint 6.8.2.

The prior changes have updated various things for new things the
linter picked up.

As noted inline, the jinja2 parsing has not been working well with
this repository.  I've been keeping an eye on it over several
releases, but I think at this point it's not going to work well for
us.  I've left discussion in .ansible-lint.

The var-spacing in stage-output is the old name for the jinja2
filtering; since that is now ignored in .ansible-lint it can be
removed.

Change-Id: Ia2c9392eeb232b9b2b1d4febce8493d71de64482
2022-11-07 10:37:53 +11:00
Ian Wienand 8c6512107c
Standarise block/when ordering
Newer ansbile-lint finds "when" or "become" statements that are at the
end of blocks.  Ordering these before the block seems like a very
logical thing to do, as we read from top-to-bottom so it's good to see
if the block will execute or not.

This is a no-op, and just moves the places the newer linter found.

Change-Id: If4d1dc4343ea2575c64510e1829c3fe02d6c273f
2022-11-07 10:37:53 +11:00
Tony Breeds 4293d8b8d9 [docs][trivial] Remove redundant colon :
Using "For Example::" marks the next block as code, so we end up
emiting ".. code-block:: yaml"[1] as a code block which we don't really
want.  Remove the additional ':' so we emit the expected HTML.

[1] Check https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-stage-output
    before merge

Change-Id: Ic2e1fb9acb6a6b4ec77bf1ee0ec9ac5d809dfb7c
2022-10-19 12:10:50 -05:00
Clark Boylan 6ff47d56db Clarify that stage-output doesn't actually write to zuul_output_dir
The stage-output role's README indicates that it stages to the
zuul_output_dir on the remote node, but this isn't true. It actually
stages to {{ stage_dir }} which has a different default than
{{ zuul_output_dir }} in fetch-output.

Update the readme to make this more clear and reduce confusion.

Change-Id: I7a7ef801db8194a7101a8dc0a6d89e1292b3fa86
2022-09-26 18:28:58 +00:00
James E. Blair 9e22cfdb0f Remove shebang from all python ansible modules
This commit in Ansible:
9142be2f6c

now allows Python modules to specify their interpreter with the shebang.
We expect our roles to use the discovered python interpreter on remote
nodes, and on the executor, we need them to use the virtualenv.  Removing
the specific shebang accomplishes this under Ansible 6, and has no effect
under older versions of Ansible.

Without this, for example, the log upload roles would not have access to
their cloud libraries.

Also update our ansible/cli check in our module files.  Many of our modules
can be run from the command line for ease of testing, but the check that we
perform to determine if the module is being invoked from the command line
or Ansible fails on Ansible 5.  Update it to a check that should work in
all 4 versions of Ansible that Zuul uses.

Change-Id: I4e6e85156459cca032e6c3e1d8a9284be919ccca
2022-09-15 13:48:10 -07:00
Clark Boylan e05b55fa67 Speed up log file fetching tasks
This replaces ansible loop tasks which rename log files to have a .txt
suffix with an ansible module task. The reason for this is each ansible
loop iteration can be very slow; up to 3 seconds per task. When
many log files need to be renamed this adds up quickly. The module can
do all of the renames in a single python process that should be much
quicker.

Note the old interface expected a list of extensions to rename. It then
converted that list of extensions into a regex used to find the files to
rename. The new code does not use regexes. It is possible someone abused
the old interface to hijack the regex and do something extra fun, but
that was not documented nor likely to be safe.

Change-Id: I7033a862d7e42b3acd8ad264aefca50685b317ff
2022-09-06 16:17:53 -07:00
Ian Wienand 5a2c7bff96 linters: update to ansible-lint 6
This updates to ansible-lint 6.  Some prior changes have updated for
the bulk of the rules, a couple of noqa points are added here.  Some
updates to the skipped rules are added.  This should have no
operational change, the only updates are cosmetic.

Change-Id: I165677bbb904f92292df00f7b9b27f8f3573aeb0
2022-07-27 17:13:39 +10:00
Ian Wienand 6d23d20f2f linters: add names to blocks
This is preparation for a later version of ansbile-lint, which finds
missing names on blocks.  This seems a reasonable rule, and the
Ansible manual says [1]

  Names for blocks have been available since Ansible 2.3. We recommend
  using names in all tasks, within blocks or elsewhere, for better
  visibility into the tasks being executed when you run the playbook.

This simply adds a name tag for blocks that are missing it.  This
should have no operational change, but allows us to update the linter
in a follow-on change.

[1] https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html

Change-Id: I92ed4616775650aced352bc9088a07e919f1a25f
2022-07-27 17:13:39 +10:00
Ian Wienand f76cfbab11 linters: fix spaces between filters
This fixes a number of places where we do not have spaces between
filters.  I think that this is a reasonable rule for readability (I
also think it probably was enforced, but maybe later versions got
better at detecting it?).

These are detected by a later version of Ansible lint; this change
should have no operational change to any roles but prepares us to
update in a follow-on change.

Change-Id: I07e1a109b87adce86f483d14d7e02fcecb8313d5
2022-07-27 17:13:39 +10:00
James E. Blair f0d7c980b5 Handle no-sudo in stage-output
The stage-output role had two assumptions:

1) The zuul_copy_output variable would always be defined
2) The role would be able to sudo on the remote node

To make it easier for users who want to use this in a base job,
remove both of those assumptions.

1) We now supply a default empty dict if the variable is not defined
so that the role does not fail with an error.

2) We check to see if we can sudo on the remote node, and if we can,
we do when copying files; otherwise we don't, and assume that the
user will only specify files they have access to (if they don't then
the copy will fail).

Change-Id: I5428c44adfafac4872342b59a92de311f41687b6
2021-05-31 15:47:57 -07:00
Sorin Sbarnea 90d263941a More E208
Change-Id: I8c3d224682d7d6780d6ad7823e390c47114e702b
2020-11-05 08:33:28 +00:00
Jens Harbott 3d86c71723 Fix with_* loop for ansible 2.9
Ansible changed handling loop variables in 2.9 [0]. Use explicit
loop statements instead of with_*.

[0] https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#migrating-to-loop

Change-Id: Ic063adbb672c3ff6bb4cca346d689630a7a37f34
2020-06-18 15:33:12 +02:00
Sorin Sbarnea 4af438d136 Made sequence indent consistent
Change-Id: I5ea4232ca4fd6e03d5b5f72eb6704bee84d04ea5
2020-05-07 14:13:55 +01:00
Albin Vass dd8e831379 use zj_log_file instead of item as loop_var
Change-Id: Ife5560ad028e293f8bfa71756fe6825c1a3f43ce
2020-05-05 18:43:30 +00:00
Albin Vass bee0c6ae2f ansible-lint: use matchplay instead of matchtask
For some reason matchtask doesn't match includes, matchplay does so use
that instead.

Change-Id: I040f7f3394503e92d06c05e8ff671a43b14baebc
2020-05-05 20:42:38 +02:00
Albin Vass 61c4eeaffd Distinguish zj_source and zj_result and item
The inner item needs to be zj_source. Call the outer zj_result
for clarity.

Change-Id: Ic5a7b18e1dba71d8dffa171246b52c81e30c6d8f
2020-05-01 15:39:21 -05:00
Albin Vass 9062289151 Check for loop_control in with_ type loops
Change-Id: I191265df7709a6262b44a428d78fe28ffaeb4b75
2020-05-01 13:45:34 +02:00
Clark Boylan a6f2ca827e Stop compressing files during intermediate steps part 2
Stop compressing files that would otherwise be handled uncompressed in
intermediate log handling. The reason for this is final log/artifact
upload should decide if a file need to be compressed or not as some file
hosting services (like swift) may not return the expected data if we get
this wrong.

This was noticed when we used swift to store intermediate tar.gz
tarballs before final upload to permanent storage. When retrieving the
tar.gz files from swift we got them back as uncompressed .tar files
(because swift was being helpful) and then later when we try to
uncompress these files we break.

Instead we'll let the upload to swift (or other storage) decide if files
should be compressed further than their existing state.

This is the second change in the stack and update the stage-output role.

Change-Id: I2fa6f123d23568ba1537a7c26d2672f04110c35b
2020-01-08 09:24:45 -08:00
Jeremy Stanley 2711d10186 Be consistent about spaces before and after vars
With the arrival of ansible-lint 4, Jinja2 variable expansions must
include spaces before and after the variable name inside the
brackets.

Adjust the new violations accordingly and remove the rule
206 exclusion.

Change-Id: Ib3ff7b0233a5d5cf99772f9c2adc81861cf34ffa
2019-08-07 20:17:41 +01:00
Luigi Toscano d89f67aadf stage-output: fix the archiving of all files
The archival task currently relies on the content of all_sources,
but some files are renamed afterwards if their extensions match
one of the special extension of the list.
This means that few files from all_sources are not found and then
not compressed.
Change then the logic: simply discover all files inside logs
and compress all of them.

Change-Id: I7d34d7d90849736b7b842c0bdd67492816f98ebc
2019-03-22 16:05:43 +01:00
Paul Belanger a5dbe86b4b Move chdir under args for shell task
This should fix a syntax issue on ansible 2.7.

Change-Id: I006ff64c40875b8f1edd1e9f9bdfe509e2bdb295
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-03-19 19:17:01 -04:00
James E. Blair dd8c88354e Fix ownership change in stage-output
If the stage_dir/logs directory was a symlink, Ansible would refuse
to perform the chmod.  However, if we tell Ansible that it is a
directory, it's fine.  This (perhaps curiously) does not tell
Ansible to turn the symlink into a directory.  It just tells it
that everything is okay.

Change-Id: Idbd78b97b95ce50b23dda8f384e32359d31013e3
2019-02-05 09:32:02 -08:00
James E. Blair 98628f718a stage-output: Add the ability to add .txt extensions to files
So that this can more easily work with files like '/var/log/syslog',
make it so that if the user specifies "logs_txt" then the file gets
a .txt extension before being placed in the logs directory.

The existing extensions_to_txt variable doesn't work for this because
it requires an existing extension.

Change-Id: I34fea0c44030c04a5540d6b62976557143289196
2019-01-31 10:48:58 -08:00
James E. Blair 2abb8a1302 Allow stage-output to copy root-owned files
This makes it easier to tell stage-output to copy files such
as syslog from remote nodes.

Change-Id: Ica2397a50de1054f5d5ec81b54ab36fda9bdc333
2019-01-30 15:38:58 -08:00
Andreas Jaeger 9566cbe07d Fix RST formatting
Remove whitespaces from build-python-release which make
the sphinx display as "** Role Variable
**" instead of bold "Role Variables".

Fix buildset-artifacts-location to display items as list.

Fix stage-output to display the example as example - and not wrap
around.

Change-Id: I8da3ec0c507d26f47ee9bb71dc9d1b3ed9ca23a5
2018-10-13 10:36:36 +02:00
Andrea Frittoli b3615c0d97 Remove support for extensions as lists
With tempest devstack and networking-odl switched to using dict,
we can now remove support for using lists for the list of extensions.

Depends-on: https://review.openstack.org/539686/
Depends-on: https://review.openstack.org/539708/
Depends-on: https://review.openstack.org/539854/
Change-Id: I62aaa737300c3b24e416f18e4bc469e7bd46ea29
2018-04-03 16:13:52 +01:00
Colleen Murphy d1956bd9f7 Fix ansible-lint job
The 'linters' tox environment was not running on roles, it was only
running on playbooks. This change adds a command to the linters
environment to ensure all roles are linted.

Since these weren't being linted, there were some problems with them.
The first was a warning about usage of the shell module. Both of these
usages seemed appropriate, so this patch adds a skip_ansible_lint tag to
each task that was failing. The second is a warning "no action detected
in task" for zuul modules. This is due to the fact that the linter
cannot find the custom module. One option is to set the ANSIBLE_LIBRARY
path to point to the zuul ansible library directory[1], but the linter
virtualenv does not actually have zuul installed. Instead, we just
disable the linter for the failing tasks.

This also cleans up a comment in the tox file that was referring to a
nonexistent zuul job.

[1] https://github.com/willthames/ansible-review/issues/16

Change-Id: Ie49da9a09733b623bb25c5a4c8aa07eacacf4b33
2018-03-28 23:10:22 +02:00
Andrea Frittoli (andreaf) ac60bc714b Improve file renaming in stage-output
We only rename the extensions for files specified in zuul_copy_output
but we should rename files in folders specified in zuul_copy_output as
well. In addition we need to rename files or folders that start with a
'.' which would otherwise not be visible once uploaded to logs.

Change-Id: Id55ab51019d10d5ccbc6531549758ac71ad276be
2018-02-16 08:03:56 +00:00
Zuul 46fb37f8bb Merge "Change the list of extensions to a dict" 2018-02-09 07:47:46 +00:00
Andrea Frittoli (andreaf) f7fd2fce61 Change the list of extensions to a dict
The stage-output role accepts a list of extensions to be replaced as txt
so that they can be seen directly from within a web browser.
Using a list means that children jobs cannot add to the parent setting,
only overwrite. Adding support for dicts while preserving, at least for
now, support for lists - to avoid breaking existing implementations.

This reverts change 844b3333aa.

Change-Id: I5052c873f0a5da7ecaa38627e8900af86bbec2f9
Needed-By: https://review.openstack.org/539686
Needed-By: https://review.openstack.org/539708
Needed-By: https://review.openstack.org/539854
2018-02-08 18:52:13 +00:00
Zuul b16399ff4b Merge "Revert "Change the list of extensions to a dict"" 2018-02-02 19:28:48 +00:00
Andreas Jaeger 844b3333aa Revert "Change the list of extensions to a dict"
This reverts commit 146fa7b2e6.

This does not work yet as planned. Let's revert for now and test properly again.

Change-Id: Idac7e398fe4854769589411f22f7c4ce7cb363f6
2018-02-02 18:33:12 +00:00
Andrea Frittoli 36aeb95399 Add no_log to verbouse tasks
Two tasks in stage-output "Register sources" and "Set source and
destination for files and folders" spam the whole output of "stat"
to be printed to console for each and every file to be staged.
The info is printed twice and it's hardly valuable for debugging purposes.

The following task, "Build a list of source, dest dictionaries for text
file", prints a more compact list of all files and folders that are going
to be staged, which I believe is enough for debugging.

To solve this add no_log to the two tasks mentioned above.

Change-Id: Id52047c429233c6684f2fca5537674f91fb2603d
2018-02-02 14:31:22 +00:00
Andrea Frittoli (andreaf) 146fa7b2e6 Change the list of extensions to a dict
The stage-output role accepts a list of extensions to be replaced as txt
so that they can be seen directly from within a web browser.
Using a list means that children jobs cannot add to the parent setting,
only overwrite. Adding support for dicts while preserving, at least for
now, support for lists - to avoid breaking existing implementations.

Change-Id: Ifc4b2070c5844e51acb9ad5d512e9f25f5bdd7ef
2018-01-31 20:58:38 +00:00
Andrea Frittoli (andreaf) 68dec9ecc5 Dereference symlinks when staging artefacts
If an artefact is a symlink, dereference it before coying it as
just copying the link may render it broken one the staged folder
is synced to the executor.

Change-Id: I5ad4dfccbc1389da3afc53f3c866d3475e006db6
2017-12-18 16:06:41 +00:00
Andrea Frittoli (andreaf) 2ea8661801 Add compress capabilities to stage artifacts
Compress files and files in folders. Use gz by default.

Change-Id: I72796e64a65fe88d85168734ac881ee3ec2ce100
2017-11-16 12:02:00 +00:00
Andrea Frittoli (andreaf) 786c48f051 Add a generic stage-output role
This role does roughly what save_dir and save_file do in d-g
functions, only it can be used for a list of files and/or
folders. Jobs can use this role to rename artifacts and place
them in a staging folder which is then used for mass sync to
the executor.

This is typically used in post - the zuul user won't necessarily
have write access to the folders with the artifacts - so it's
easier to stage everything to a folder owned by the zuul user.

Change-Id: Id45a50821539985710da7172a9185fd91e803603
2017-11-02 07:22:26 +00:00