We use pyenv to install python3.12 on ubuntu jammy (which doesn't have
native packages for this version of python). We don't update the
container images just yet as we want to build confidence through testing
before end users see the new python version in production.
Change-Id: I20a57f6d29699886867ce912e9943c59aa9963f4
This reverts commit 1c188ad440.
Reason for revert: This change increases test failures; investigation is ongoing.
Change-Id: I3a101656ee0904e1cf6c856dba845385c333260f
I'm not sure if this is a good idea or not yet. Testing with MySQL
actually exposed the alembic transaction thing that is fixed in a
grandparent change. That said much of the world seems to be moving to
MariaDB so it might be a good idea for us to test that more explicitly?
We update bindep.txt which will install mariadb during CI job runs. We
also update our local test setup tooling to roughly match what will be
used in CI. For example the mariadb:10.6 docker image is chosen because
this is roughly equivalent to what Jammy packages.
Another good reason for this change is that Debian ARM64 does not
package MySQL and only packages MariaDB. This should give us better
compatibility for testing on architectures other than x86-64.
Change-Id: I9d00557ca5823da34278c3e9ecda2daaf13440b7
This adds python3.10 testing on Jammy and switches the docker images to
python3.10 from 3.8.
We run sudo for postgres with -Hi to avoid non fatal errors when
postres' client attempts to write command history to Zuul's homedir (it
is running as the postgres user which can't write to zuul's homedir). We
also need to update the libffi package version for jammy to 8 in
bindep.txt. Finally, python_version values need to be quoted as "3.10"
is different than 3.10 which is equivalent to 3.1 when serialized by
yaml as a float.
Force setuptools to use stdlib (shipped by the distro) distutils to
avoid problems with virtualenvs not actually being virtualenvs.
Finally we switch the bulk of jobs over to using nodeset: ubuntu-jammy
as the default python there is 3.10.
Change-Id: I97b90bb7a23c90f108f23dda9fdd0e89f9f4dbca
Having ambient capabilties causes bwrap to error on start [1]
unless the bwrap executable also has the setuid bit set or is run as
root.
This can cause issues in openshift or podman unless ambient
capabilities are dropped [2].
[1] - bae85baf72/bubblewrap.c (L742)
[2] - https://github.com/containers/bubblewrap/issues/380
Change-Id: I15455fb400448d7672638f911d6cf045fa683a9b
The container images are based on opendevorg/python-base which is
based on python which is based on debian. Upstream python image has
updated from buster to bullseye. Once that makes it through
opendevorg/python-base, our images will start being bullseye. Make
the bindep entries forward compatible with such a future.
Change-Id: I71182e9d3e6e930977a9f983b37743ee3300ec91
This mirrors the configuration in Nodepool for using TLS-enabled
ZooKeeper in tests. We use the ensure-zookeeper role in order
to get a newer ZooKeeper than is supplied in bionic.
Change-Id: I14413fccbc9a6a7a75b6233d667e2a1d2856d894
While upgrading tox37 to tox38, those lines were not updated/removed
they can be removed as:
-tox-py3X jobs specify python version to install during ensure-python
-other jobs are most likely using ensure-pip, which on ubuntu-bionic
install python3-venv and python3.6
Change-Id: I0abfa76d4a7057c8b9154cc111bb0aa20f4fe0bb
This supports the afs-related roles in zuul-jobs by supplying the
unlog and vos commands. Notably, it does not actually build and
install the afs kernel module because that is a recommended (not
required) package, and we don't automatically install recommended
packages.
Change-Id: I1a22cfa44b7eaa4dd57393e082f42843187a77d9
We added openafs-krb5 but we need krb5-user for kinit.
Add DEBIAN_FRONTEND=noninteractive to the Dockerfile
to prevent the krb5-user package from asking for our
default realm.
Change-Id: Ifbef43887e541a3edc259ffaf9a75d7343c97dca
The ps command is needed by zuul and supplied by the procps
package. This is not necessarily installed by default so require it.
Change-Id: I7d5b614ff517863d93a46789bec4065a6201d137
This updates bindep to include libyaml so that we use the C yaml parser
in testing and on our container images. This should speed up zuul in
some cases.
Change-Id: I4895f60fac248c9890ec80b03decad7973be52d5
It's already installed on rpm and apk but missing on ubuntu platforms
where it's not necessarily pre-installed so pull it in for all
platforms. This is required by fetch-sphinx-tarball of zuul-jobs so it
should be in every zuul deployment.
Change-Id: I5fb32826053ce7fe0229cea9ca89d2f7244d4044
bzip2 is universal across different distributions, it's also
needed by the 'unarchive' module when running inside the executor.
Change-Id: I24ddff86b8962c902c3e541a72e8810c163cb0c9
python-base is buster now, so this is not necessary. We can
just install bubblewrap and socat from buster directly.
Change-Id: If65f3c2d6367a7c79cf9d6d8f788021ba72cccd4
Add the correct libre2 package name for Debian buster, and also
update the quickstart playbook and documentation to deal with
the change in default rsa key encoding format from newer
versions of ssh-keygen.
Change-Id: I6ada88cd896d844c1171f7bcaf4691dea023d51f
This change adds unzip and bzip2 to the rpm platform bindep so that executor
can run the fetch-sphinx-tarball role from zuul-jobs.
Change-Id: Ibe679af11b88b0912afd6d3e35a0ef9ed3fb2c8b
As a first step towards supporting multiple ansible versions we need
tooling to manage ansible installations. This moves the installation
of ansible from the requirements.txt into zuul. This is called as a
setup hook to install the ansible versions into
<prefix>/lib/zuul/ansible. Further this tooling abstracts knowledge
that the executor must know in order to actually run the correct
version of ansible.
The actual usage of multiple ansible versions will be done in
follow-ups.
For better maintainability the ansible plugins live in
zuul/ansible/base where plugins can be kept in different versions if
necessary. For each supported ansible version there is a specific
folder that symlinks the according plugins.
Change-Id: I5ce1385245c76818777aa34230786a9dbaf723e5
Depends-On: https://review.openstack.org/623927
Alpine by default comes with busybox and its version of 'du'. This
version doesn't have the '--max-depth' switch. While this can be fixed
in zuul it further unlike the gnu version of du counts the files that
are hard linked to the cache. This makes it unsuitable for zuul and we
need to require the gnu version on alpine.
Change-Id: I39032a7ea6e7b0262eed8b1c49128e64f06029d7
This has been fixed as an upstream dependency in the python-devel
package and can be removed from our explicit bindep list. Additionally
the package was not found or required on all rpm systems.
Change-Id: I05bd688af8353df819f27437666dbb65ddda19c2
In anticipation of building containers based on python:alpine (and on
bindep understanding alpine packages) add alpine packages to bindep.txt.
We'll get testing coverage that this list works from the container build
job.
Change-Id: I0ab864adb6bf519d7fdf0c1cf1b4a01064df5322
Looking ahead towards potentially using pbrx to build containers of
zuul, we want to indicate some bindep packages are needed only for
building a wheel, not for installing/running it.
Introduce a 'compile' profile that can be used to get the distro dependencies
needed for building wheels of the python dependencies. Also add runtime
library depends without a profile so that bindep -b properly reports the
packages needed for installing if wheels have been built.
Zookeeperd is moved to the 'test' profile as it is not required to be
installed locally for zuul to operate.
Depends-On: https://review.openstack.org/580872
Change-Id: I3c8dcedddd8634661b3524966a6f228c9edde29b
Now that we have ubuntu-bionic images online, we can start gating for
python3.6.
We only need to install bubblewrap PPA for ubuntu-xenial,
ubuntu-bionic already has bubblewrap packaged.
Update bindep.txt skipping ubuntu-xenial for bubblewrap.
Change-Id: Ic850a07001a9adef9cc459fcb107e0ebaaddf8ab
Depends-On: https://review.openstack.org/565884
Depends-On: https://review.openstack.org/565948
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
The fb-re2 compile requires g++ which the gcc package doesn't appear to
pull in by default on Suse. During review it was mentioned that CentOS 7
doesn't pull it in either. Fix this by installing gcc-g++ on all rpm
platforms.
Change-Id: Iee4da053f1d760eb3e3012cc815a2967af0e415b
python3-devel package is not available on CentOS repositories and this
causes an unnecessary error to be logged to console while installing
zuul which might confuse users.
[centos@zuul zuul]$ sudo yum install $(bindep -b) -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.mirror.constant.com
* epel: mirror.cs.princeton.edu
* extras: mirror.cs.pitt.edu
* ius: mirror.ehv.weppel.nl
* updates: mirror.clarkson.edu
No package python3-devel available.
Error: Nothing to do
For CentOS 7, the proposed zuul-from-scratch document instructs users to
install python35u-devel package from ius-release repo.
https://review.openstack.org/#/c/564948/
Change-Id: I9df218b56c59df196eff6cbd5183884c39d1ab7b
The github status requirements matching and trigger filter are
currently plain text matching based. This currently limits sharing of
pipeline definitions between tenants as zuul reports the status as
'<tenant>/<pipeline>'. This currently makes it necessary to define
trigger filter for each tenant [1] and completely blocks pipeline
requirements.
A solution to this is regex matching which makes it possible to define
the filter once [2].
Further this enables an interesting further use case to trigger on any
successfull status [3]. This makes it easier to cooperate with other
CI systems or github apps which also set a status.
Directly use re2 as this will be used in the future for regex
matching.
[1] Trigger filter snippet
trigger:
github:
- event: pull_request
action: status
status:
- zuul:tenant1/check:success
- zuul:tenant2/check:success
- zuul:tenant3/check:success
- zuul:tenant4/check:success
[2] Regex trigger filter snippet
trigger:
github:
- event: pull_request
action: status
status:
- zuul:.+/check:success
[3] Generic success filter snippet
trigger:
github:
- event: pull_request
action: status
status:
- .*:success
Change-Id: Id1b9d7334db78d0f13db33d47a80ffdb65f921df
Some deployments use postgres as database backend. In order to have
official support for that we need to add tests for it.
Change-Id: I36408c27ccb36fd0826d97f34d98b211ee075f4e
Depends-On: https://review.openstack.org/535718
We no longer support python2.7 for zuul, we can be more specific
and now only install python3 libraries.
Change-Id: Ic31221cb55a09d9568a9f5f26c412c0559555f01
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
We need graphviz for docs building, use "doc" tag in bindep for it.
This is needed with updated jobs that look for doc tags.
Change-Id: Idc329ab113761061678c9bd31485ac5e18273e23
This is needed on Fedora systems to build python packages with
gcc, but for some reason is not listed as a python-devel dep.
Change-Id: Ia7479154867902b816603dc2babf6b08b91ca919
Bubblewrap is a setuid wrapper for unprivileged chroot and namespace
manipulation. We'll be using this to run ansible-playbook from.
Change-Id: I7c6414281724d066812094d02a726df357978076
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Create an interface to the cryptography library so that internally
Zuul uses simple facade methods. Unit test that interface, and
that it is compatible with OpenSSL.
Change-Id: I57da1081c8d43b0b44af5967d075908459c91687
Every project should have a public and private key to encrypt secrets.
Zuul expects them to already exist under /var/lib/zuul/keys on the
scheduler host. If an operator manages these keys externally, they
should simply be placed there. If they are not found, Zuul will
create them on startup and store them there so they will be found on
the next run.
The test framework uses a pre-generated keypair most of the time to
save time, however, a test is added to ensure that the auto-generate
code path is run.
Co-Authored-By: James E. Blair <jeblair@redhat.com>
Change-Id: Iedf7ce6ca97fab2a8b800158ed1561e45899bc51
This will allow us to enter results from all jobs for
use with the openstack-health dashboard.
Depends-On: I08dbbb64b3daba915a94e455f75eef61ab392852
Change-Id: I28056d84a3f6abcd8d9038a91a6c9a3902142f90
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Add a requirement on kazoo and add a Zookeeper chroot to the test
infrastructure.
This is based on similar code in Nodepool.
Change-Id: Ic05386aac284c5542721fa3dcb1cd1c8e52d4a1f
Zuul doesn't have a database, so there's no need to install one, let
alone two, in its bindep.txt file. Remove them.
Change-Id: Ib1ac968d06d059d6184a8fec031cf198d0439596
Zuul doesn't have a database, so there's no need to install one, let
alone two, in its bindep.txt file. Remove them.
Change-Id: Ib1ac968d06d059d6184a8fec031cf198d0439596
This should fix the error we're seeing in gate jobs:
ValueError: jpeg is required unless explicitly disabled
using --disable-jpeg, aborting
Change-Id: I686b21e452aad2dfb9358360137070f58dde3882
Clark Boylan