The ubuntu-xenial labels are going to disappear from opendev as that
image is EOL and will we deleted. Clean up our zuul config.
Update some example reference as well.
Change-Id: Id04110f7c871caa1739ff2b62e9796be4fb9aa00
This change add a new lib/host-mem file and moves the existing
ksm support to a new configure_ksm function.
Additional support for ksmtuned is added with a new flag
"ENABLE_KSMTUNED" which defaults to true.
This change also adds support for zswap. zswap is disabled
by default. When enabled on ubuntu lz4 will
be used as the default compressor and z3fold as the zpool.
On non debian distros the compressor and zpool are not set.
The default values should result in very low overhead although
the zstd compressor may provide better overall performance in ci
or with slow io due to the higher compression ratio.
Additionally memory and network sysctl tunings are optionally applied
to defer writes, prefer swapping and optimise tcp connection
startup and keepalive. The sysctl tunings are disabled by default
The base devstack job has been modifed to enable zram and sysctl
tuning.
Both ksm and zswap are wrapped by a tune_host function
which is now called very early in devstack to ensure
they are configured before any memory/network intensive
operations are executed.
The ci jobs do not enable this functionality by default.
To use this functionaltiy define
ENABLE_SYSCTL_MEM_TUNING: true
ENABLE_SYSCTL_NET_TUNING: true
ENABLE_ZSWAP: true
in the devstack_localrc section of the job vars.
Change-Id: Ia5202d5a9903492a4c18b50ea8d12bd91cc9f135
This reverts commit 67630d4c52.
Reason for revert: Seeing random failures across jobs as sometimes
'keyring_pass.cfg' gets duplicated keys and that makes executions
of any openstackclient command to fail until the file is removed.
This should be handled before re enabling the token caching again.
Change-Id: I3d2fe53a2e7552ac6304c30aa2fe5be33d77df53
Related-Bug: #2042943
SDK uses python keyring library to enable token caching. Normally this
is requiring a proper desktop (interactive) session, but there are some
backend plugins working in non-interactive mode. Store cache in an
unencrypted file on FS (this is not worse than storing passwords in
plaintext).
Change-Id: I42d698f15db5918443073fff8f27b926126d1d0f
Since we are python3 only for openstack we create a single python3
virtualenv to install all the packages into. This gives us the benefits
of installing into a virtualenv while still ensuring coinstallability.
This is a major change and will likely break many things.
There are several reasons for this. The change that started this effort
was pip stopped uninstalling packages which used distutils to generate
their package installation. Many distro packages do this which meant
that pip installed packages and distro packages could not coexist in the
global install space. More recently git has made pip installing repos as
root more difficult due to file ownership concerns.
Currently the switch to the global venv is optional, but if we go down
this path we should very quickly remove the old global installation
method as it has only caused us problems.
Major hurdles we have to get over are convincing rootwrap to trust
binaries in the virtualenvs (so you'll notice we update rootwrap
configs).
Some distros still have issues, keep them using the old setup for now.
Depends-On: https://review.opendev.org/c/openstack/grenade/+/880266
Co-Authored-By: Dr. Jens Harbott <frickler@offenerstapel.de>
Change-Id: If9bc7ba45522189d03f19b86cb681bb150ee2f25
This patch: https://review.opendev.org/c/openstack/devstack/+/882299
provides functionality, that commit hash can be passed as last arugment,
however when GIT_DEPTH is set, it fails, as in:
timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git ./ovn
--depth 1 --branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
fatal: Remote branch 03b95a4566a15f7544f4cdf35629dacede4dcf55
not found in upstream origin
Closes-Bug: #2023020
Change-Id: I748354964a133e028e12458cc9014d6d014cbdb9
git_clone assumes a branch or a tag is passed as the last argument, and
it fails when a commit hash is passed, as in:
timeout -s SIGINT 0 git clone https://github.com/ovn-org/ovn.git
/opt/stack/ovn --branch 36e3ab9b47e93af0599a818e9d6b2930e49473f0
Cloning into '/opt/stack/ovn'...
fatal: Remote branch 36e3ab9b47e93af0599a818e9d6b2930e49473f0 not found
in upstream origin
Change-Id: Id1328d7cba418fa7c227ae9db4fe83c09fd06035
devstack-base is changed to descend from
openstack-multinode-fips which is defined in
project-config.
This allows jobs to execute the enable_fips playbook
to enable FIPS mode on the node, but only if they
opt-in by setting enable_fips to True. Otherwise,
this is a no-op.
Change-Id: I5631281662dbd18056ffba291290ed0978ab937e
This patch includes changes required to run devstack on RHEL 9.
- en_US.utf8 is provided by glibc-langpack-en
- iptables command is provided by iptables-nft
- Use /etc/os-release to identify the distro in RHEL 9 as it doesn't
provide lsb_release command.
- CRB repository name is different from CentOS 9
Change-Id: I8f6d9263b24f9c2cf82e09258e2d14d7766ad337
We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc
openEuler 20.03 LTS SP2 support was removed from devstack in last
few months due to its python version is too old and the CI job
always fail. And openEuler 20.03 LTS SP2 was out of maintainer in May
2022 by openEuler community.
The newest LTS version was released in March 2022 called 22.03 LTS.
This release will be maintained for at least 2 years. And the python
version is 3.9 which works well for devstack.
This Patch add the openEuler distro support back. And add the related
CI job to make sure its works well.
Change-Id: I99c99d08b4a44d3dc644bd2e56b5ae7f7ee44210
Because adding the role is idempotent, we can save doing the initial
check for role assignment. Also simplify the output matching by using
osc's filters where appropriate.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: If2a661cc565a43a7821b8f0a10edd97de08eb911
Similar to other functions, this uses "--or-show" to avoid double
calls.
Co-Authored-By: Jens Harbott <harbott@osism.tech>
Change-Id: I548f9acd812687838e04b705f86f3b70d2b10caf
Right now we don't officialy support LinuxMint as our
documentation says [1], it seems LinuxMint is a relict
and got forgotten over time.
This patch removes LinuxMint from the code in order not to
confuse users.
[1] https://docs.openstack.org/devstack/latest/
Closes-Bug: #1983427
Change-Id: Ie1ced25f89389494b28a7b2e9bb1c4273e002dd5
This patch reduces memory usage on the Cinder Volume and Backup services
by tuning glibc.
The specific tuning consist on disabling the per thread arenas and
disabling dynamic thresholds.
The Cinder Backup service suffers from high water mark memory usage and
uses excessive memory. As an example just after 10 restore operations
the service uses almost 1GB of RAM and does not ever free it afterwards.
With this patch the memory consumption of the service is reduced down to
almost 130MB. If we add a revert from Cinder (Change-Id
I43a20c8687f12bc52b014611cc6977c4c3ca212c) it goes down to 100MB during
my tests.
This glibc tuning is not applied to all Python services because I
haven't done proper testings on them and at first glance they don't seem
to have such great improvements.
Related-bug: #1908805
Change-Id: Ic9030d01468b3189350f83b04a8d1d346c489d3c
Currently, neutron tunnel endpoints must be IPv4 addresses,
i.e. $HOST_IP, although IPv6 endpoints are supported by most
drivers.
Create a TUNNEL_IP_VERSION variable to choose which host IP
to use, either HOST_IP or HOST_IPV6, and configure it in the
OVS and Linuxbridge agent driver files. The default is still
IPv4, but it can be over-ridden by specifying TUNNEL_ENDPOINT_IP
accordingly.
This behaves similar to the SERVICE_IP_VERSION option, which
can either be set to 4 or 6, but not 4+6 - the tunnel overhead
should be consistent on all systems in order not to have MTU
issues.
Must set the ML2 overlay_ip_version config option to match
else agent tunnel sync RPC will not work.
Must set the OVN external_ids:ovn-encap-ip config option to
the correct address.
Updated 'devstack-ipv6-only' job definition and verification role
that will set all services and tunnels to use IPv6 addresses.
Closes-bug: #1619476
Change-Id: I6034278dfc17b55d7863bc4db541bbdaa983a686
The job is broken since it is running with python3.7 and most services
now require at least python3.8.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ie21f71acffabd78c79e2b141951ccf30a5c06445
This updates each devstack service library, to use it as the
default value for service-specific RBAC configuration.
Change-Id: I41061d042206c411ee3dd94ce91098e612af7ae7
This is necessary for more consistent behavior across multiple
distro versions. Apparently somewhere along the way, git started
looking at the current user's home directory instead of $HOME.
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
Change-Id: I941ef5ea90970a0901236afe81c551aaf24ac1d8
git commit [1] introduced a new behaviour to work around a CVE that
disallows any git operations in directories not owned by the current
user.
This may seem unrelated to installation, but it plays havoc with PBR,
which calls out to git to get to get revision history. So if you are
"pip install"-ing from a source tree you don't own, the PBR git calls
in that tree now fail and the install blows up.
This plays havoc with our model. Firstly, we checkout all code as
"stack" then install it globally with "sudo" (i.e. root) -- which
breaks. We also have cases of essentially the opposite -- checkouts
we have installed as root, but then run tox in them as a regular user;
tox wants to install the source in its venv but now we have another
user conflict.
This uses the only available configuration option to avoid that by
globally setting the source directories we clone as safe. This is an
encroachment of the global system for sure, but is about the only
switch available at the moment. For discussion of other approaches,
see [2].
Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798
[1] 8959555cee
[2] https://review.opendev.org/c/openstack/devstack/+/837636
Change-Id: Ib9896a99b6d6c4d359ee412743ce30512b3c4fb7
After patch [1] new RBAC policies changed in the way that SYSTEM_ADMIN
user isn't anymore allowed to e.g. create resources in behalf of some
projects. Now PROJECT_ADMIN needs to create such resources instead.
So this patch basically reverts most of the changes which were done
in [2] some time ago.
It also introduces new entry in the clouds.yaml file -
"devstack-admin-demo" which is "admin" user in the "demo" project as
it's needed to create some resouces in the demo project now.
Additionally, because of bug [3] this patch changes way how IPv6
external gateway IP is found using Neutron API. This change may be
reverted in the future when bug [3] will be fixed.
[1] https://review.opendev.org/c/openstack/neutron/+/821208
[2] https://review.opendev.org/c/openstack/devstack/+/797450
[3] https://bugs.launchpad.net/neutron/+bug/1959332
Depends-On: https://review.opendev.org/c/openstack/neutron/+/826828
Closes-Bug: #1959196
Change-Id: I32a6e8b9b59269a8699644b563657363425f7174
This is needed so we can set keystone into enforcing secure RBAC.
This also adjusts lib/glance, which already partially used
devstack-system-admin.
Change-Id: I6df8ad23a3077a8420340167a748ae23ad094962
openEuler is an open-source Linux based operating system. The current
openEuler kernel is based on Linux and supports multi arch, such as X86_64
and aarch64. It fully unleashes the potential of computing chips. As an
efficient, stable, and secure open-source OS built by global open-source
contributors, openEuler applies to database, big data, cloud computing,
and AI scenarios. openEuler is using RPM for package management.
Note:
Currently there is no available package for uwsgi-plugin-python3 and ovn, so that
openEuler needs manually install them from source.
Website: https://www.openeuler.org/en/
Change-Id: I169a0017998054604a63ac6c177d0f43f8a32ba6
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
This patch includes changes required to run devstack on CentOS Stream 9
which has been already published in official repos by CentOS team [1]:
- Add RDO deps repository for CS9.
- remove xinetd package from installation for swift. Note that
rsync-daemon is installed which should work fine.
- Replace genisoimage by xorriso in CS9.
- Use /etc/os-release to identify the distro in CS9 as it doesn't
provide lsb_release command.
- Use pip from rpm package instead of from get-pip.py as done in Fedora.
- Add non-voting job devstack-platform-centos-9-stream to the check
pipeline.
Change-Id: Ic67cddabd5069211dc0611994b8b8360bcd61bef
This commit formalizes some additional users to act as different project
users and updates clouds.yaml file so they're easy to use.
It creates:
- a reader on the demo project
- a reader on the alt_demo project
- a member on the alt_demo project
With the adoption of secure RBAC personas, these are useful for using
OpenStack APIs as that work continues.
Change-Id: I3237a771275311377313b7d7d80ac059ac69d031
Keystone has supported system-scope since Queens and we already make
sure we create a cloud profile for system-admin in
/etc/openstack/clouds.yaml.
This commit ensures keystone creates a couple of new users to model
system-member and system-reader personas. Doing this by default in
devstack makes it easier for people to use.
We've already taken a similar approach in tempest by setting up the
various system personas for tempest clients to use.
Change-Id: Iceb7c5f517db20072e121dc7538abaa888423c67
If ERROR_ON_CLONE is set to True which is case for
all the devstack based job, devstack does not clone the
repo instead raise error. From current error message, it
is difficult to know that ERROR_ON_CLONE is True until we
traceback the code or check devstack-base job set ERROR_ON_CLONE
to True.
Current error message is like:
-------
+ functions-common:git_clone:560 : echo
'The /opt/stack/oslo.limit project was not found; if this is a gate job, add'
The /opt/stack/oslo.limit project was not found; if this is a gate job, add
+ functions-common:git_clone:561 : echo 'the project to the
$PROJECTS variable in the job definition.'
the project to the $PROJECTS variable in the job definition.
+ functions-common:git_clone:562 : die 562 'Cloning not
allowed in this configuration'
--------
Adding ERROR_ON_CLONE info in error message will help to
know the reason of devstack not cloning the repo.
Change-Id: I9e9852f046fefb299b4ef4446323e9c86437212f
The Secure RBAC effort has updated Ironic such that it
can support a mode where it is scope enforcing for all
interactions with the API. Due to the design, and operating
nature of Ironic's API, services speaking with it must
authenticate with a system scope to have a full picture
of the universe.
In this case, we need to update the nova configuration
accordingly such that the compute service understands
how to talk to ironic so that it can see the nodes under
management.
Ironic will likely update this again at a later point in
time to enable a "hybrid" mixed-mode as the operating model
and related permissions *should* allow nova to use a project
scoped "owner" account with Ironic, in order to access
and command nodes to deploy. But at this time, we're
focusing on the exclusive operating mode.
Change-Id: I1946725ce08c495178c419eaf38829f921c91bbe
Needed-By: https://review.opendev.org/c/openstack/ironic/+/778957
The XenAPI driver was removed during the Victoria release [1], while the
libvirt+xen driver has been removed in the Wallaby release [2]. Remove
references to Xen from DevStack since its all a no-op now.
[1] I42b302afbb1cfede7a0f7b16485a596cd70baf17
[2] I73305e82da5d8da548961b801a8e75fb0e8c4cf1
Change-Id: If7055feb88391f496a5e5e4c72008bf0050c5356
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
When deploying on the centos 8 stream variant
the output of "lsb_release -i -s" is
CentOSStream instead of CentOS
This breaks the is_fedora function in devstack
preventing package installation and removal.
Change-Id: I39ccefbd06f46adf5077f8d8001f37d3b190f040
This function has been deprecated for a long time, let's finally
remove it. It is only generating a warning anyway.
Change-Id: I7bd440adf2ce8283e3ad3d5d09e6b2b877e2b42e
Previously pyc files were only cleaned if clean.sh was run.
with this change a new clean_pyc_files function was introduced
with the logic that was previously in clean.sh but it is now
invoked from unstack.sh
With the previous behavior you could not stack with horizon
enabled then unstack and stack again due to the presence of pyc
files that were owned by root.
By moving the clean to unstack in stead of clean.sh you can
now stack, unstack and stack again without hitting the pyc issue.
since unstack is invoked by clean the existing clean.sh behavior has
not changed in practice except for the fact the pyc files are
removed sooner in the process.
This change also removes support for findutils < 4.2.3
Ubuntu 12.04 and CentOS 6 both have 4.4.2 since they were
released 8 years ago and are now EOL its fair to assume
that all modern distros have 4.2.3+
https://repology.org/project/findutils/versions
Change-Id: I13c9aad9be7e0930a0d875b7d382090caf0b8982
There are a few dependencies which are really os-brick-specific.
They are listed in its bindep.txt file, but os-brick is usually
installed from pip, so its bindep.txt file is not available.
As those dependencies are needed by the various services
which use os-brick, move them to their own dependency file
(with the addition of the new lsscsi, required by the next
os-brick stable release) and make sure that file is parsed
when installing the services which require os-brick.
Side note: there should be a way to avoid this duplication
also for pip-installed libraries (normal services can use
files/ or even bindep, but in this case the source is not always
available), (temporarily?) duplicate them, as it has been
the case for the other os-brick dependencies already listed here.
Change-Id: I9ab6e215dbef9ebdb1946da2f9a40ce020ecc95b
Dr. Jens Harbott
yatin