Commit Graph

125 Commits

Author SHA1 Message Date
Jeremy Stanley f87307b86a Pin ansible-compat<4 until we upgrade ansible-lint
There was a regression introduced in ansible-compat v4 which stopped
ansible-lint from working. This was fixed in ansible-lint 6.16.0 but
we've pinned it for so long now that there are a lot of rule
violations to fix or mark for skipping. In the interim, pin back all
new versions of ansible-compat until someone has time to fix things
or rip out ansible-lint entirely.

Change-Id: I68e06bbfaf0b38582320f09659dfc462016eacf8
2023-05-16 20:05:54 +00:00
Zuul c2aa277faa Merge "Add a convenient normalize-gerrit-acl tox testenv" 2023-05-10 15:25:00 +00:00
Jeremy Stanley 1832dcdad7 Add a convenient normalize-gerrit-acl tox testenv
Users struggle with Gerrit ACL formatting, often going through 4 or
5 revisions to get things like indents, trailing whitespace,
alphabetical ordering and final newlines correct. With this testenv,
users can run something like this to auto-reformat their Gerrit
ACLs...

  tox -e normalize-gerrit-acl -- gerrit/acls/foo/bar.config

We'll include this as a suggestion in the Infra Manual section on
adding projects as well.

Change-Id: I5f0903849ea72f9d1c81164bed4c651c9423f1fc
2023-05-10 00:29:25 +00:00
Jeremy Stanley b0bca88203 linters: yet still more broken ansible-compat
This avoids another broken release of ansible-compat per the linked
bug (assuming the next version of ansible-lint will fix it, so
optimistically not using <), and is a continuation of the workaround
we merged for the prior version.

Change-Id: Id2369dc9a16a85981fd9b7122976f3b6120891f2
2023-05-05 19:52:56 +00:00
Jeremy Stanley 9e9dfbb576 linters: avoid broken ansible-compat
This avoids a broken release of ansible-compat per the linked bug
(assuming the next version will fix it, so optimistically not using
<), and is the same workaround we merged to the
openstack/openstack-zuul-jobs project in
https://review.opendev.org/881938 earlier today.

Change-Id: Ia43e8e481e34320bca2bed4a32b780f997c24c85
2023-05-02 12:43:24 +00:00
Ghanshyam Mann 6facccc00e Fix tox4 error
With tox4 in use for testing, tox
run failing with passenv value and
external commands run.

failed with pass_env values cannot contain
whitespace, use comma to have multiple values in a single
line, invalid values found 'http_proxy HTTP_PROXY https_proxy
HTTPS_PROXY no_proxy NO_PROXY' [tox/session/cmd/run/single.py:54]

Fixing passenv value and adding external commands
via allowlist_externals.

Change-Id: I05d36dab6895dfc14d026230592def51b78bb755
2022-12-24 20:20:21 -08:00
Clark Boylan f7af758c33 Update tox.ini for tox v4
This convers whitelist_externals to allowlist_externals to make tox v4
happy. Note that this may not be a complete fix for tox v4 as
project-config isn't tested speculatively which makes it harder to find
all the cases.

Change-Id: Ia44a2ef83cdc57d0bb74c72ea5c95078b53ccaec
2022-12-12 10:29:20 -08:00
Jeremy Stanley 109c7dbc23 Correct inline comment about ansible-lint pin
It's https://github.com/ansible/ansible-lint/issues/2320 which is
blocking our use of newer ansible-lint (though it's worth revisiting
whether testing with ansible-lint is still of any real value).

Change-Id: Ie3e42df7f90e5349bf1513e8896b1d5e16125b42
2022-09-07 18:43:21 +00:00
Jeremy Stanley ddfdf914a1 Match the ansible-lint <6.5 pin from zuul-jobs
We try to test our job content with the same ansible-lint versions
used by the zuul/zuul-jobs repository. Adjust our exclusions to
match what it's using, for consistency and ease of future
maintenance.

Change-Id: I2300e8950780245a38f68ce9cc5aa5ff62e65e40
2022-08-29 23:25:01 +00:00
Jeremy Stanley a87b6d69f5 Exclude .cache when running flake8
For local runs of testenv:linters we cache all the external Ansible
playbooks and roles ansible-lint will need to reference in order to
validate content in this repository. We build that cache in a .cache
directory, but not everything in the cached repositories is suitable
for flake8 checking, so add that directory to its exclusion list.

Also add the same directory to .gitignore, so Git will stop
complaining about it being untracked.

Change-Id: I1f740edd75cc7f6ba0f4a73f0a409a620b437b16
2022-08-26 19:51:57 +00:00
Tony Breeds 46838fbda8 Exclude ansible-lint 6.5.1 also.
ansible-lint introduced a regression as described in
"Don't use ansible-lint 6.5.0" I1770e034772b55b993d276d580c5a75d059bd822
the fix is still in draft and 6.5.1 has been released so lets skip that
too

Change-Id: I641a1246f870869153df782381fd9dafa180da26
2022-08-25 19:46:39 -07:00
Jeremy Stanley c9c122460e Don't use ansible-lint 6.5.0
We have some skips in our playbooks which 6.5.0 ignores because of
https://github.com/ansible/ansible-lint/issues/2320 so block that
version and hope the fix makes it into the next release.

Change-Id: I1770e034772b55b993d276d580c5a75d059bd822
2022-08-24 18:26:41 +00:00
Ian Wienand 826a79f50f linter: update ansible-lint; add auto-download of roles
This updates ansible-lint to the 6.x releases.  It also updates the
ansible installed to our current zuul version.

It cleans up the config file and marks it as yaml.  A few new
exceptions are added with explainations.

We don't need to fake the zuul_return by telling ansible-lint to mock
it.  This is added to the config, and we can remove the stub file.

A constant source of problems running this locally is that you have to
have the other role repos checked out so ansible-lint can do its
checks.  Add a smaller helper script for doing this locally.  In the
gate, ANSIBLE_ROLES_PATH is set to the Zuul checkouts of these
projects.  Locally, add a smaller helper "ansible-lint-roles-cache.sh"
that pulls the projects into a .cache directory.  If they are already
there, they get updated.  By default locally we will use these
checkouts.  This way, "tox -e linters" just works without having to do
anything else.

This also modifies the xargs to run the check all at once, instead of
fork for each file.  I did try autodetection but it seems like other
yaml files in the roles/playbook directories still confuse
ansible-lint.

Also I don't think we need a ansible-playbook --syntax-check step;
ansible-lint covers that.

Change-Id: I972f73037b9f904a555b81f3835ca5261639ed01
2022-08-12 09:29:11 +10:00
Neil Hanlon 75789007a5
remove basepython override for grafyaml
grafyaml requires slugify which had a release yesterday. The py3 wheel
  is being installed in the tox env, leading to Syntax errors

Change-Id: I2d577d7cc78e752ad29001e09d7e5134cfc106f7
2022-02-17 08:51:24 -05:00
Clark Boylan e99d3cb407 Update ansible-lint to fix the rich dep
ansible-lint is currently broken when older than a recent release. This
is due to to the 'rich' dependency updating without a pin and not
working with older ansible.

Change-Id: Id93a2bc0b452bafca1dca8550ffc825d72533cdc
2022-01-10 11:30:00 -08:00
Dr. Jens Harbott 6ca813f6c8
Fix project-config testing
This is a combination of 2 commits:

Fix the renaming of ansible-role-refstack-client

In [0] that project was moved from x/ to openinfra/, but the patch
missed that at the same time refstack was also moved, changing the
path for the ACL.[1]

[0] https://review.opendev.org/c/openstack/project-config/+/765787
[1] https://review.opendev.org/c/openstack/project-config/+/808479

Fix use of yaml.load()

The use of this function has been deprecated for a long time[2]. With
PyYAML==6.0 the call is now failing, so replace it with the safe
version.

[2] https://msg.pyyaml.org/load

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I439d1f167b5893a134a5b628ebc00fd6617e0cbd
2021-10-18 13:23:29 +02:00
Jeremy Stanley ef03e7fa36 Drop use of track-upstream
When upgrading from Gerrit 2.13 to 3.2 we stopped relying on a local
fork of its source and have been building from (more recently
completely unmolested) upstream source. This fork was the only place
we were relying on jeepyb's track-upstream feature, so we can stop
looking for it in our checks and normalization as well.

Remove the check_gerrit_projects_changed scripts as well, as they
only existed to run things where track-upstream was enabled.

Change-Id: I597c1a577c53e2db61413d9ec531378667691d2a
2021-07-06 16:45:22 +00:00
Jeremy Stanley e858068643 Switch the IRC access check to OFTC
Make some adjustments to the IRC access check script so that it
works in the OFTC network now. Also update the channel config to
reflect the new ACL paradigms there. Remove our volunteer operators
temporarily until we can confirm their nicks there. Also rip out the
channel forwarding for unregistered nicks, we can work on readding
it there later if we determine it's necessary after all, but it will
need implementing differently anyway if so.

Change-Id: Ib3c43ef5ba22191d869629cd01d3800f3e235ea4
2021-05-27 17:09:04 +00:00
Ian Wienand a531d83bac tox.ini : update Ansible pin
ansible-lint is incompatible with Ansible 2.7 which is now causing
failures.  Update to current Zuul versions.

Change-Id: I65878a896281ad67d512f0bd9219aa2058e4953a
2020-11-05 10:32:31 +11:00
Andreas Jaeger 01dc9e1039 Fix ansible-lint
The ansible-lint testing fails silently with:
2020-09-23 20:08:26.289902 | ubuntu-bionic | [2425] /home/zuul/src/opendev.org/openstack/project-config$ /bin/bash -c 'find roles -maxdepth 1 -mindepth 1 -type d -printf "%%p/\n" | xargs -t -n1 ansible-lint'
2020-09-23 20:08:26.294180 | ubuntu-bionic | ansible-lint %p/
2020-09-23 20:08:26.908714 | ubuntu-bionic | WARNING  Couldn't open %p - No such file or directory [try:0]
2020-09-23 20:08:27.910102 | ubuntu-bionic | WARNING  Couldn't open %p - No such file or directory [try:1]
2020-09-23 20:08:28.911658 | ubuntu-bionic | WARNING  Couldn't open %p - No such file or directory [try:2]

Fix invocation.

Change-Id: I5d701506c82701970f818520672c2d4ff1885629
2020-09-23 22:11:39 +02:00
Monty Taylor 0a4e2aab9a Don't install python-dev on focal
Also, install yamllint in the dib env, as it's a requirement
of dib-lint now but is only in test-requirements for dib.

Change-Id: I083bca901ca51438099d1d3bbbd0076ac3d7da07
2020-05-27 10:24:51 -05:00
Clark Boylan e25845d8fa Escape use of % in tox.ini to avoid interpolation
Zuul-jobs tox roles wants to be able to run tox --showconfig -e linters.
Unfortunately our linters target has an unescaped % symbol in it which
causes python3 + tox to fail with an interpolation error. Escaping the %
to avoid interpolation addresses this.

Change-Id: I7e23595495909fc94a7ea5a9e461e67e25ddeae9
2020-05-19 15:15:05 -07:00
Ian Wienand 0326363e48 Update ansible-lint and add fake zuul_return
- Use zuul_return fake module to make ansible lint happy, this allows
  to remove Zuul import.
- While ansible-lint 4.2.0 is now able to detect playbooks/roles, this
  is broken, so don't use it
- move its config out of tox.ini so it can be used by any tools, or
  without tox

Change-Id: Ie8935f47db855647e19ae091044e5ac1871f1551
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Co-Authored-By: Andreas Jaeger <aj@suse.com>
2020-03-18 22:28:10 +01:00
Ian Wienand 4f6629021e Update hacking, fix errors/warnings
This version of hacking doesn't understand f-strings as usable in
Python 3.  Update to the latest and fix current issues, which are all
just formatting fixes.

Change-Id: I0a7d6f93f07477b6dd29ab143130dd9064c250be
2020-01-14 09:40:45 +11:00
Andreas Jaeger 800ea0aa05 Switch project-config to promote jobs [3]: Update building
For using opendev-tox-docs, jobs need to publish to a "html"
subdirectory, update the docs-site and specs jobs.

Update .gitignore for this.

Update python files to use yaml.safe_load to avoid a warning.

Remove unused docs environment from tox.ini.

Depends-On: https://review.opendev.org/682213
Change-Id: I8d7032a6e836ec26f347d16abd600514089a677b
2019-09-14 17:03:01 +02:00
Zuul 5fe966e6a6 Merge "Update to ansible 2.7 for linting" 2019-06-02 16:15:48 +00:00
Andreas Jaeger 7114e79485 Check ACLs for all namespaces
Remove hardcoded list of namespaces with iterating over all directories
in the check of valid ACLs.

Remove parameter, we don't need to pass it in.

Be more verbose and report directories scanned and number of ACL files
read.

Change-Id: I4193ac59b431d31f3ce835c93950745e5b742418
2019-05-31 21:53:03 +02:00
Zuul bd06273806 Merge "Update hacking to version 1.1" 2019-05-20 01:00:09 +00:00
Andreas Jaeger 5e11aa650c Update to ansible 2.7 for linting
We run ansible 2.7 with zuul, so update the version we use for linting.

Change-Id: Ifdbd75272e525b499fe97cbf72e9d735fd13876e
2019-05-17 16:35:43 +02:00
Andreas Jaeger 10de11d7a9 Check zuul/main.yaml projects
Check that projects in zuul/main.yaml exist in gerrit.
This would have avoided https://review.opendev.org/659697.

Use yaml.safe_load in one case.

Depends-On: https://review.opendev.org/659735
Change-Id: I6fb3d90c18b57c004caf02dd6e5d5c34b1ce1ff9
2019-05-17 11:12:45 +02:00
Andreas Jaeger ecc2997769 Update hacking to version 1.1
Update hacking, fix a couple of problems found.

Change-Id: Ie903e3ddf13e78d758b26e884bb1fb0e1a6ed73c
2019-05-17 09:31:30 +02:00
Monty Taylor 72b6395c6b Update links for opendev
Change-Id: I35c8c64abfed31daf2be3698be13e72e38cd321b
2019-04-21 14:01:51 +00:00
Andreas Jaeger c0972b00b0 Fix propose-project-config-update
After the recent change, the job now fails since ruamel.yaml is not
installed. Change the job to use tox to install dependency.

This creates a new tox environment that is then called from the proposal
script.

Failure log:
http://logs.openstack.org/periodic/git.openstack.org/openstack-infra/project-config/master/propose-project-config-update/bf68ac6/

Change-Id: I02ce33f10d388531806380f464dacc5617c74f1f
2019-04-11 14:09:58 +02:00
Andreas Jaeger e7f77f283f Some python yaml fixes
libyaml now needs to Loader keyword to not throw a warning, use
safe_load instead everywhere.

sort_keys isn't supported for reasons surpassing understanding, so
switch to ruamel.yaml for the places where we write contents back
out.

Our ruamel.yaml wrapper strips extra leading space, which is needed
when the root object is a list. When the root object is NOT a list,
it's not needed. Add a flag to the constructor to toggle the
behavior.

For gerrit/projects.yaml this leads to a different wrapping of
description fields, which is included here.

Let ruamel.yaml handle comments

Since we've switched to ruamel.yaml, we don't need to strip
comments. Stop stripping them. Intra-list comments aren't
supported, so just remove them.

Please don't ask me why we can't have a space before openstack-merlin.

Change-Id: I72776774eabd4e397ee5e8bdd58b3632be8fd3a1
2019-04-10 13:33:51 +00:00
Ian Wienand 74540cbb58 Replace openstack.org git:// URLs with https://
This is a mechanically generated change to replace openstack.org
git:// URLs with https:// equivalents.

This is in aid of a planned future move of the git hosting
infrastructure to a self-hosted instance of gitea (https://gitea.io),
which does not support the git wire protocol at this stage.

This update should result in no functional change.

For more information see the thread at

 http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003825.html

Change-Id: I0cc16c32dcea3b6036553c9f9e8df45a241254e0
2019-03-24 20:35:36 +00:00
James E. Blair d8bbee54aa Update zuul ansible location
With multi-ansible support, the location of the zuul plugins/modules
has moved.

Also, handle the sort_keys argument which is new in pyyaml 5.1.
Set it to False to achieve the old behavior.

Also use python3 by default for tox tests on this repo (note,
one env still explicitly requests 2.7).

Change-Id: I8e2cf1f2276bca5a912fed52062628b3b9d0d6d9
2019-03-18 12:02:25 -07:00
Paul Belanger be36dbb8c3 Set ANSIBLE_ACTION_PLUGINS for linters
The zuul_return plugin was recently converted to an action plugin, make
sure we also export that path for ansible-lint.

Change-Id: Id74ccb58cb436ce40bd643b6def7b1db4a280c46
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-01-18 17:29:24 -05:00
Monty Taylor 0d274e1a88
Pin ansible-lint <4 in the linters venv
Change-Id: I1440cab79e248b533454a7b8e9eec9916c02b6b0
2019-01-05 01:06:58 +00:00
Monty Taylor 755117184d
Stop installing nodepool in all the test envs
We only need it in the nodepool env. Also, install from pip and let
tox-siblings take care of updating from source for us.

Depends-On: https://review.openstack.org/611629
Change-Id: I0ece7b751662b4198276e88adcf263634fd792d6
2018-10-19 13:34:26 -05:00
Monty Taylor b90a9c5ee8
Update ansible library location to envsitepackagesdir
We install zuul so that we can pick up library entries. The proper path
for this is envsitepackagesdir. Also, just list zuul in the requirements
so that we're not cloning remotely. tox-siblings will take care of it in
the gate for us, because we add zuul to required-projects in the
openstack-zuul-jobs-linters job.

This adds a setup.py and setup.cfg so that tox-siblings can do its
thing.

Depends-On: https://review.openstack.org/611607
Change-Id: I864308af3424a0b0505a252d0d5cf44277172fbf
2018-10-19 13:34:23 -05:00
Zuul 754d78a875 Merge "Bump ansible for linters" 2018-08-13 16:33:01 +00:00
Ian Wienand f78146bac6 linters: Take a guess at the roles path
It's probably reasonable to assume that for interactive use, the
required roles for linting in zuul-jobs and openstack-zuul-jobs are in
sibling directories of project-config.  Set this as the default
environment value.

Change-Id: I388fdae998ac3cb87c30616ed761c888e1e429f0
2018-08-10 14:30:06 +10:00
Ian Wienand 4b30fa5635 Bump ansible for linters
I noticed this when I tried to use Ansible 2.5 "loop:" constructs.

Unfortunately, we can't just rely on bringing in zuul to pull the
right version of ansible.  ansible-lint being uncapped just takes it
over, and we end up with the latest version.  Pin it to the current
zuul requirements and add a note (maybe one day
https://github.com/pypa/pip/issues/988 will get a fix ...)

Change-Id: Iaf8f0f5cdc46df41fec7c436c1179c80bb5c368e
2018-08-10 14:15:04 +10:00
Monty Taylor 68c355d346
Add linters check to make sure site-variables is yaml
A YAML syntax error in site-variables was merged recently. Errors in
that file break basically everything, so add a check that the file at
least parses.

Change-Id: I9b2c47a0461275e1d53aabe387f0309719d46b77
2018-07-27 11:46:39 -04:00
Ian Wienand ee2efa10e1 Use tox for grafyaml validation
Currently this calls out to grafyaml-validate.sh which tries to
install zuul-cloner and then pull grafyaml from that.

Since zuulv3 doesn't *have* zuul-cloner, this means the job is broken
for local testing.  It works in the gate only because zuul-cloner
compatability shims are installed.

To maintain the testing-against-master behaviour, the dependent change
provides the grafyaml source in the environment; otherwise we go to
git directly.

Depends-On: https://review.openstack.org/572251
Change-Id: I3d5cd062414b77aaa70d2669730fda870ca0168e
2018-06-05 13:58:16 +10:00
Colleen Murphy 2fd1a55e63 Fix ansible-lint job
The 'linters' tox environment was not running on roles, it was only
running on playbooks. This change adds a command to the linters
environment to ensure all roles are linted. This also cleans up a
comment in the tox file that was referring to a nonexistent zuul job.

Change-Id: I04d9d79fca16a7f78f9c646f96e5fba78a5f5329
2018-03-29 15:35:19 +02:00
Paul Belanger 40fab3dafc
Switch to python3 for tox -enodepool
Now that nodepool only support python3, best to test against it.

Change-Id: Iaa85c6f7f9d64b03c62e80437822735b6a0fa085
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-01-25 15:35:57 -05:00
Andreas Jaeger 60128af709 Test all nodepool files
We now have nl03, loop over all nodepool files so that new files get
automatically tested.

Change-Id: Iddd4afda10a6cb95b7783ab0454c805d8df7313e
2018-01-19 19:41:23 +01:00
Andreas Jaeger 052a1ee648 Partial revert "Revert "Remove diskimages section from nodepool-launchers""
Add back in tox.ini changes to test all versions of nodepool.yaml files.

This reverts partially commit 7c4da0c8b8.

Also, test nodepool master branch instead of deleted feature/zuulv3
branch.

Change-Id: I0b04d251f62efc362a80a7c6fb595e76c4bf3d76
2018-01-19 09:05:04 +01:00
Zuul 6adbc07012 Merge "Revert "Remove diskimages section from nodepool-launchers"" 2018-01-18 21:44:28 +00:00