Commit Graph

49 Commits

Author SHA1 Message Date
Takashi Kajinami 8f183d1671 Periodically update Puppetfile_unit
This updates the script to automatically update puppet constraints so
that Puppetfile_unit is updated as well as Puppetfile. We introduced
this separate file[1] a while ago but have not been updated it
properly. Updating it periodically helps us catch up with any update
in the dependent modules more timely.

[1] 047ea125ddefa8151d6592d4772560a1b23a13dc

Depends-on: https://review.opendev.org/875301
Change-Id: I4d89a2985781f14d9ff7f76f412f0b8ccc4c6ddb
2023-02-26 01:48:56 +09:00
Dr. Jens Harbott 702037af47 Switch the requirements-constraints job to py310
Since the PTI switched the supported python versions for the current
cycle from 3.8+3.9 to 3.9+3.10, we want to update the constraints
generation accordingly. Since we don't have a stable platform where we
can run both py3.9 and py3.10 in parallel, generate constraints only for
py3.10 for now.

Change-Id: I4a4d9cb6e292bb693ae424be0928501fed81113f
2022-10-12 05:38:30 +00:00
Clark Boylan 55215ac11c Set ensure_pip_virtualenv_command in Focal proposal jobs
Previously we had this error:

  No key/value pairs provided, at least one is required for this action to succeed

Which appears to be due to the default(omit) condition when setting
ensure_pip_virtualenv_command in focal proposal jobs. The issue is
default(omit) doesn't do what you think it does when defining a
variable. It is really only useful for defining module parameters that
may be omitted.

Instead we need to call ensure-tox two different ways depending on
whether or not we want to override the defaults for
ensure_pip_virtualenv_command. One method for Bionic and another for
Focal keyed off of whether or not _venv_command is defined.

Change-Id: I0cbca64f4a31c8b4eacb5e1c50f2e9fb289ce18e
2022-07-27 09:08:04 -07:00
Ian Wienand 4616695be0 proposal updates: fix typo in release match
This fixes a typo in I683a61b1bae1a809caf724aee87a21af2e18fb0c -- it
should be "ansible_distribution_release"

Change-Id: I074f089371054856a8b96bd6e650496a0bd53685
2022-07-27 14:49:09 +10:00
Ian Wienand 6f1e9fb2b9 translation updates: refactor Python 3.8 workarounds
I think we have quite a tangle of fixes on fixes in the translation
jobs, and I hope this refactor makes things a bit better.

The fundamental problem seems to be that python3.6 can't work with
master requirements.txt.  But we can not move some jobs off bionic
(python 3.6) because of java version contsraints.

The first thing that happened was these jobs got "ensure-python" with
"python_version: 3.8" set to bring in python3.8 on Bionic.  This role
was not bringing in python3.8-venv, so we had no way to create 3.8
virtual environments.  Thus ensure-pip ended up getting modified to
drag this package in (reverted with notes in
372f3af706c796331578ef81c2680e8c4c03c3b6).  This has a fairly easy
solution; let's just make ensure-python bring in the venv packages for
the udpated python -- see dependency
Ie3c03fea82bcec80a897f0905c15f35405a50396.

The git-review install here uses 'ensure_pip_virtualenv_command' to
install git-review in a virtualenv.  git-review still supports bionic,
and does not need to be installed in any speical way.  However, by
overriding the 'ensure_pip_virtualenv_command' for the entire job,
this unrelated install now grew a dependency on python3.8 and created
more confusion over why it was failing.

I believe what we actually want to do on Bionic is just run tox under
python3.8.  So this pulls out all the job-level defintions and
encapsulates them in a single block scoped to run on bionic.  It will
pull in python3.8 packages, and then install tox under python3.8 with
an override of the virtulenv command just for that step.  The jobs
just maintain their node definition and note on why it is required.

The long-term future of these jobs is obviously a question; but I
believe this keeps them running on bionic in as sane a way as possible
for now.

The second part of this is the python3.9 interpreter required on
Focal.  Setting "python_version: 3.9" in the job definition is a bit
confusing, because the job actually requires both Python 3.8 AND 3.9.
This variable is passed to ensure-python, and the naming is a bit
unfortunate because it is so broad (but we can't change that without
zuul-jobs updates).  Here I've pulled out the Python 3.9 install into
a separate focal-gated section in the pre.yaml that explains what is
going on (note that Python 3.8 is just the base python on focal, so
needs no extra installation).  I've also updated the description on
the job node with this info.  There's no need to install the tox
environment any differently on focal; it just uses the base system
python (3.8) which is fine.

Depends-On: https://review.opendev.org/c/zuul/zuul-jobs/+/850957
Change-Id: I547e5eb7faabcd9d3983ae85f3291b1a740bc77c
2022-07-27 07:48:43 +10:00
yatinkarel be4b276729 Only install python3.8-venv in translation update jobs
Currently both python3.8 and python3.9 venv package
is being installed, but since python3.9-venv is not
available in bionic, python3.8-venv installation also
fails as mentioned in [1].
This patch moves ensure_pip_from_upstream_interpreters
var from proposal pre playbook to job definition and
override it in translation jobs to attempt installation
only for python3.8-venv.

Also switch back to "python3.8 -m venv" virtualenv_cmd
as with this patch python3.8-venv will get installed.

[1] https://review.opendev.org/c/zuul/zuul-jobs/+/846201
[2] https://review.opendev.org/c/openstack/project-config/+/846390

Change-Id: I9d9a423e3147d3f04c1951a796636e5520202a6d
2022-06-20 22:50:41 +05:30
yatinkarel 0ef308725f [proposal-updates] Run ensure-python before git-review setup
Install git-review task relies on ensure_pip_virtualenv_command
variable but with [1] this is set to python3.8 in translation
update jobs but it get's installed later by ensure-python role,
let's run it earlier to fix the issue.

[1] https://review.opendev.org/c/openstack/project-config/+/845907

Change-Id: Iecbdcb50063cb461b3a09ddfa205d071c7d77b75
2022-06-16 11:37:05 +05:30
Dr. Jens Harbott c7d42980c6 Fix propose-updates job for requirements (3rd attempt)
The previous attempts at fixing the job were not successful yet, it
seems the variable in the pre playbook isn't overridden by a job
variable. Since the other uses of this job are running with ubuntu-focal
anyway, set the list of available python interpreters in the playbook
directly and remove the modifications from the requirements specific job
again. Also pin the nodeset for this job to guard against changes in the
base job, which will likely require another change to the interpreter
list.

[0] I9735f34639777c928fbe06febc16dd24667b1405
[1] I52cc316ce9756c4411c18c8339d44c457cb59897

Change-Id: Iddd61e1be766f2bb6da4bf62766829247fa143ce
2022-05-26 23:00:13 +02:00
Matthew Thode 91bbf1fdc6
update generate constraints to py38,39
3.10 will come later when there is an image that supports all three

Change-Id: I663de65a971337fce65a51f0e5a7f959b8da2c03
Signed-off-by: Matthew Thode <mthode@mthode.org>
2022-04-14 10:58:33 -05:00
Sean McGinnis 848d080c1a
update-constraints: Install pip for all versions
Depends-on: https://review.opendev.org/739272

Change-Id: Ic6444e6c186c82a21e7fe2868c27fda3378c187a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-07-10 08:55:50 -05:00
Sean McGinnis 118a7ba3d4
Make tox global for update proposal jobs
If00afb50c37b45aefbb45070da4efef3e43e62b2 updated the requirements
update job to use tox as part of the update checking script. We have
ensure-tox as one of the roles in the pre playbook for this job, but it
is not set to be made available globally, which results in the script
failing with being unable to find tox.

This updates the role inclusion to set ensure_global_symlinks: true so
tox will be available in the script.

Change-Id: I5d3ed4ce62cd8df604802fedf522d644d5664698
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-06-24 10:34:18 -05:00
Sean McGinnis bfc06ae553
Fix venv use in requirements propose_update.sh
This script creates a virtualenv to run a requirements script to update
constraints. virtualenv is not globally available anymore. Rather than
trying to call it directly, this now uses tox to set up the virtual
environment.

Change-Id: If00afb50c37b45aefbb45070da4efef3e43e62b2
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-06-23 07:51:44 -05:00
Sean McGinnis 9c91006304 Ensure pip is installed for propose-update-constraints
Pip is used by this task, make sure ensure-pip is used so it is
present.

While here, install git-review to a venv and symlink it globally to
keep things separate.

Change-Id: If6b1abd2545c68f727bddc5317797236f451b67c
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Signed-off-by: Ian Wienand <iwienand@redhat.com>
2020-06-17 16:02:46 +10:00
Clark Boylan 8ee0a3a869 Install git-review under python3 for proposed updates
We were installing git-review under python2 in order to propose updates
to Gerrit. git-review happily runs under python3 so update the
installation to avoid python2.

Change-Id: I1671345d71cd60f6508a2c067c4e308680234afe
2020-06-10 15:12:34 -07:00
Andreas Jaeger b59a59da35 Use python 3 venv for generate-constraints
With change I15d451f6a832b2e698b28e7351a36ef7aea92abe
generate-constraints needs python 3, setup the venv using python 3.

Change-Id: I3dba35267dc6f344ad27910c7dc18c1a33abf4ba
2020-06-07 08:39:19 +02:00
Zuul 2e9c2c6fd0 Merge "install python3.8 now that victoria is in development" 2020-05-17 18:59:57 +00:00
Matthew Thode dc799bd092
install python3.8 now that victoria is in development
Change-Id: I7758b3879215f8977be82bba273e771d84cd3155
2020-05-17 13:32:40 -05:00
Matthew Thode 5b69bb5edd
drop python2.7 from generate-constraints
Change-Id: I5e51b46afbc26332368525f61a84ff9bd0536a1e
2020-05-15 18:36:42 -05:00
Matthew Thode 45ed2e8fa8
add py38 to constraint bot so victoria work can begin
after requirements branches we will remove py27 and py37

Change-Id: I5fde605634108dd628df546a2318731e1832b6b3
Signed-off-by: Matthew Thode <mthode@mthode.org>
2020-04-12 11:24:02 -05:00
Ian Wienand 0326363e48 Update ansible-lint and add fake zuul_return
- Use zuul_return fake module to make ansible lint happy, this allows
  to remove Zuul import.
- While ansible-lint 4.2.0 is now able to detect playbooks/roles, this
  is broken, so don't use it
- move its config out of tox.ini so it can be used by any tools, or
  without tox

Change-Id: Ie8935f47db855647e19ae091044e5ac1871f1551
Co-Authored-By: Sorin Sbarnea <ssbarnea@redhat.com>
Co-Authored-By: Andreas Jaeger <aj@suse.com>
2020-03-18 22:28:10 +01:00
Matthew Thode 23d169ebf9
stop generating constraints for py3.4 and py3.5
https://governance.openstack.org/tc/goals/selected/train/python3-updates.html

Change-Id: I87806f0e17e678c048035213ceebb3e073161d55
2020-02-06 10:52:00 -06:00
Mohammed Naser e8afe9128f proposal: add sync for doc/* in openstack-ansible
We have started to maintain some of the items inside doc/* to be
sync'd such as requirements to build docs, this patch adds it.

Change-Id: I73c32ea17a9fb0f6d0ba1887f8618aa508ba63c5
2019-07-19 13:42:51 -04:00
Matthew Thode 24254e5db1
add version map for python3.6 to 3.7
This prevents installing unconstrainted versions of libraries for
python3.7.  When it was unmapped there was no constraint for it for
certian libraries (some libraries had diferent versions for each version
of python).

Change-Id: I0a5ecc715142756a9ec37fca559fad691651430c
2019-05-03 15:49:04 -05:00
Andreas Jaeger 14eb58501b Fix nodepool after OpenDev changes
Fix a couple of places where orgs where renamed so that dib finds the
repos.

Change-Id: Ic52dcffcd9323c12410e05b1a3598911a28cbcaa
2019-04-21 17:05:34 +00:00
Andreas Jaeger ee6468b9b4 Update jobs for OpenDev
Follow OpenDev changes:
* Use gerrit.opendev.org instead of gerrit.openstack.org
* Use opendev.org insted of git.openstack.org
* Follow move of openstack-infra/project-config to
  openstack/project-config

Change-Id: Ic27ea41407abc0bdec29768e5ee5dca33f8adeda
2019-04-20 13:51:05 +00:00
yatin 1cb8c2bcf4 [generate_puppetfile] Add support for handling commit
We sometimes [1] need to use a commit hash as a pin,
so adapt the script to work with commit hash pins as well.

[1] https://review.openstack.org/#/c/586946/

Change-Id: I8867b3d41dc6ceef084c6d978cc405b9f6710375
2018-07-30 20:00:03 +05:30
Tony Breeds 9620683a66 Run propose-updates for requirements-constraints to Python3.6
Currently we're running propose-updates forrequirements-constraints
under python 3.5 (on xenial).  We take the results of freezing the
constraints fro 3.5 and emit them as 3.4 constraints also.  In the near
term[1] we'd like to use the same technique to generate 3.6 constraints.

With the inclusion of I8784509bc162eb6f2e80261bc2d81dbe63ce7989, and
knowing that we no longer run on trusty node, we can now clone the 3.5
constraints freeze to both 3.4 and 3.6.  However given that soon we'll
be moving from xenial to bionic this introduces debt that will need to
be fixed after the August release.

Instead just move propose-updates for requirements-constraints job to
bionic now and start cloning 3.6 to 3.5 and 3.4.

Several jobs use propose-updates as a parent so I've made this change
specifically where we call the propose-updates job specifically for
requirements-constraints

Note: This may feel very familiar if you reviewed
I541b64afaee3bf466ff1a461fd2cb472657c4986, The aim is the same but I
mis-understood what propose-update-constraints was doing and it wasn't
the right job and was reverted in I75280b23e2e60800af45dc2343c82008ec9971a9

[1] See re-design discussion:
    http://lists.openstack.org/pipermail/openstack-dev/2018-July/132152.html

Change-Id: I1874d4f20a6d69a881f0d9407e159c35ab21542d
2018-07-25 08:55:47 +10:00
Tony Breeds cde92dd039 Revert "Run propose-update-constraints under Python3.6"
This reverts commit c1de884d33.

propose-updates-constraints != propose-updates.  The later does what I want, the former is used by the release-team after releasing a new library.

#eggonface

Change-Id: I75280b23e2e60800af45dc2343c82008ec9971a9
2018-07-24 22:43:11 +00:00
Tony Breeds c1de884d33 Run propose-update-constraints under Python3.6
Currently we're running propose-update-constraints under python 3.5 (on
xenial).  We take the results of freezing the constraints fro 3.5 and
emit them as 3.4 constraints also.  In the near term[1] we'd like to use
the same technique to generate 3.6 constraints also.

With the inclusion of I8784509bc162eb6f2e80261bc2d81dbe63ce7989, and
knowing that we no longer run on trusty node, we can now clone the 3.5
constraints freeze to both 3.4 and 3.6.  However given that soon we'll
be moving from xenial to bionic this introduces debt that will need to
be fixed after the August release.

Instead just move propose-update-constraints job to bionic now and start
cloning 3.6 to 3.5 and 3.4.

[1] See re-design discussion:
    http://lists.openstack.org/pipermail/openstack-dev/2018-July/132152.html

Change-Id: I541b64afaee3bf466ff1a461fd2cb472657c4986
2018-07-17 10:01:09 +10:00
Zuul 7f44cb0db8 Merge "proposal/OSA: Sync the contents of the sync folder" 2018-07-11 13:55:43 +00:00
Zuul 6cde7a034d Merge "Revert "proposal: OSA: Do not sync tests/tests-repo-clone.sh anymore"" 2018-07-11 13:55:41 +00:00
Jesse Pretorius 18e54ff4c7 proposal/OSA: Sync the contents of the sync folder
We have an emerging need to synchronise a common set of tasks across
roles. While it may sometimes make sense to create a role for this
sort of thing, the overhead of doing so would be greater than simply
copying the file between roles. We're also finding include_role to
be somewhat buggy, especially when used from inside a role.

This patch adds the ability to copy a set of files from the source [1]
to the same path (without the 'sync' prefix) in the destination roles,
abiding by the same rules as all other synchronised files.

[1] openstack/openstack-ansible/tests/sync
Relates-to: https://review.openstack.org/579527
Change-Id: I598ff6aae50e6570d5f262c830c4be10f1cc0b75
2018-07-06 15:46:43 +00:00
Jesse Pretorius (odyssey4me) 6bd28acf8f Revert "proposal: OSA: Do not sync tests/tests-repo-clone.sh anymore"
This reverts commit 022703d084.

We actually still want to sync this file for stable branches, so we revert
this commit.

Change-Id: I63ac677b620cd627b6a4c8dfa6c8e978d4ec6f61
2018-07-06 14:58:50 +00:00
Doug Hellmann e7d25bb989 remove "requirements" mode from propose_update.sh
We do not automatically propose updates to projects' requirements, so
we don't need this branch of logic any more.

Depends-On: https://review.openstack.org/555402/
Change-Id: I511ead56a6a6efecde25ce6b118256aa0560023a
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-05-07 21:26:01 +00:00
Markos Chandras 022703d084 proposal: OSA: Do not sync tests/tests-repo-clone.sh anymore
The tests/tests-repo-clone.sh file is being phased out from the OSA
repositories so the proposal bot should not sync that anymore.

Change-Id: Ia923fe60a80110ea09a350670c54a4011a13b089
2018-03-28 08:53:46 +01:00
Andreas Jaeger fbc1f58c30 Move release-tools out of jenkins/scripts
To not install these on the images, create a new scripts directory to
hold them.

Create new copy-release-tools-scripts role to copy the files over,
we do not need using legacy-copy-project-config-scripts anymore since
everything is copied elsewhere.

Add the role to all playbooks that use release-tools scripts.

Change-Id: I36c458e297fec58023799f625739b341c4d403cf
2018-02-27 19:27:35 +01:00
Andreas Jaeger 3cbae461ad Install proposal scripts only when needed
Move proposal scripts from jenkins/scripts to the proposal directory
so that we can install them only for proposal scripts and do not need
to bake them into nodes.

Change-Id: I356906e4a43597cc913bfa539cac7c2c6dfce117
2018-02-15 08:04:24 +01:00
Andreas Jaeger affdbab218 Consolidate common.sh script copying
Create new role for copying common scripts, this includes for now only
common.sh. Use this for proposal and translation jobs.

In the past we duplicated the file, this allows to have one definition
only.

Change-Id: I72c14341f511fdaa2c67a3758387afc9445869cc
2018-02-15 08:03:40 +01:00
Monty Taylor 09da196701 Remove explicit bindep parameters
All of these invocations of the bindep role are passing what amount to
the defaults. In zuul-jobs, doing bindep_dir: "{{ zuul_work_dir }}" is
important because it allows in-repo uses of those jobs to override how
they work for things like triggering from one repo but running on
another.

The publication jobs in project-config do not need that level of
flexibility, so just remove params and let defaults take over.

As a follow up, we could also add defaults for zuul_work_dir - and
honestly for bindep_dir - to the base job. That way we can always count
on them being set to a sane default value, and they're still
overrideable in a job.

Change-Id: I5bcda84aefa4726d0a904e8bc9698c21ca07f9a5
2018-01-05 22:17:17 +01:00
Doug Hellmann ae2422adf6
move git configuration for release jobs to ansible tasks
Instead of scripting the git configuration, use tasks in pre playbooks
for the jobs. We use different settings for tag-releases and
propose-constraint-updates.

Change-Id: Id3713d89a8b2af821de41a146954be551337cf75
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2017-10-20 18:38:12 -04:00
Andreas Jaeger 7fed969131 Run bindep for proposal jobs
We need gettext on the systems for the translations jobs, this should be
part of the bindep setup. Since we invoke tox, add bindep and tox setup
in pre-run.

Also, some proposal jobs need pyyaml, so add python-yaml to bindep.txt -
and use bindep for all proposal jobs.

Change-Id: I3c0e4741e1c0186b78859841db1ca5c791d4806b
2017-10-20 12:52:47 +02:00
Andreas Jaeger bf61f85cf8 Fix REFNAME in propose_update.sh
ZUUL_REFNAME is not set in periodic pipelines.

Provide optionally branch and use that one instead if REFNAME is not set
to mimic how the script was used in v2.

Change-Id: I3a514b5ffd9d0797c1bd43a0762bd5709630b1cd
2017-10-17 20:18:07 +02:00
Monty Taylor 1167acf702
It is pre_tasks, not pre-tasks
Change-Id: Id3991cba84b1d2044eb756b981a862abe193e83a
2017-10-16 08:22:32 -05:00
Monty Taylor 13877e739e
Use propose-updates for requirements-constraints
We don't need a legacy job for this, we have a propose-updates job that
can do any of the tagets the script knows about.

We do need to add git-review to the remote host though, so do that.

Change-Id: I3cf10120ab0e9aff65c35468d3a7bb9d42725712
2017-10-15 10:11:14 -05:00
Monty Taylor 6cef6fdbb3
Use zuul_legacy_vars for propose jobs for now
We need to rework this completely to be v3 native. For now, just grab
the legacy vars.

Change-Id: I49613839b96b72ced4cb6b1733ef3fa6531dcb66
2017-10-10 23:12:15 -05:00
Monty Taylor 57c7899cf0 Make project-config gating in zuul v3
So that we can land a changes to project-config after v2 has been turned
off, shift project-config to v3 and remove from v2.

Change-Id: I67fc88a86d4e8b094cab88929bb6b8c47198964a
Depends-On: I7cfb90df4eb49defce7aa41453918347a2c6742e
2017-09-26 10:40:22 -07:00
Monty Taylor a2ccdf2d08
Extract scripts copying into a role
We'll need it for more than just proposal jobs, but it's clearly very
project-config specific, so keep it here.

While refactoring that, consume add-sshkey from zuul-jobs.

Depends-On: I569b93ba0e775896b5a4f59b544fe32d1e2dce9b
Change-Id: I6b16acdcf55bcc6ff28ad496c044fe7db4599cd8
2017-08-28 11:11:06 -05:00
Monty Taylor abee455990
Add a run playbook to propose-updates
By adding a run here, we can easily make jobs either as children or
variants since the only thing needed is the argument.

Change-Id: I6d19b8d9fb96210b8b92103c095fd8c8b9c8309b
2017-08-25 17:34:32 -05:00
Monty Taylor cb1fe3a91d
Add propose-update-constraints job
After we release to PyPI, we want to send out update-constraints.

We don't need to rewrite all the scripts involved with this for now, so
update them to call each other using relative not absolute paths.

Then, copy the scripts dir from the trusted checkout on the executor
to the homedir on the remote host so that we can execute the appropriate
script. Local paths are relative to the location of the playbook.

Change-Id: I080f71fefe79f7faa581fae91eac74384cef4050
2017-08-25 17:33:56 -05:00