Commit Graph

1130 Commits

Author SHA1 Message Date
Ghanshyam Mann 1fe7707cf0 Ignore 500 status code in generate plugin script
Due to various reasons, this script may encounter the
500 status code from some repo (x/fuel-plugin-onos in current case[1])
If that happen then it return failure status code to the
propose-updates job and fail that job

- https://zuul.openstack.org/builds?job_name=propose-updates&project=openstack%2Fdevstack&skip=0

It is better not to raise the 500 error in this script and just ignore those
repo to process further to detect the plugin.

[1] https://zuul.openstack.org/build/dba0aa41d145472397916dfcd13948de/log/job-output.txt#2442

Change-Id: Ibca0a2aac404161340e8fc00170018eecf5c8326
2024-03-05 08:37:03 -08:00
yatin 82c30cd82e Revert "Enable keystone token caching by OSC"
This reverts commit 67630d4c52.

Reason for revert: Seeing random failures across jobs as sometimes
'keyring_pass.cfg' gets duplicated keys and that makes executions
of any openstackclient command to fail until the file is removed.
This should be handled before re enabling the token caching again.

Change-Id: I3d2fe53a2e7552ac6304c30aa2fe5be33d77df53
Related-Bug: #2042943
2023-11-15 12:44:50 +00:00
Artem Goncharov 67630d4c52 Enable keystone token caching by OSC
SDK uses python keyring library to enable token caching. Normally this
is requiring a proper desktop (interactive) session, but there are some
backend plugins working in non-interactive mode. Store cache in an
unencrypted file on FS (this is not worse than storing passwords in
plaintext).

Change-Id: I42d698f15db5918443073fff8f27b926126d1d0f
2023-10-26 11:07:30 +02:00
Zuul bd472fd53d Merge "Revert "Woraround systemd issue on CentOS 9-stream"" 2023-08-22 07:00:11 +00:00
yatin 7c4a955c52 Revert "Woraround systemd issue on CentOS 9-stream"
This reverts commit 113689ee46.

Reason for revert: systemd-252-17.el9 which includes the fix is now available in CentOS 9-stream repos.

Change-Id: I6fe19838a75a30fd5d2434c03b7f403f1c7e4b50
2023-08-21 06:28:30 +00:00
Zuul c1d1954e91 Merge "Add option to install everything in global venvs" 2023-08-11 18:04:15 +00:00
yatinkarel 113689ee46 Woraround systemd issue on CentOS 9-stream
systemd-252-16.el9 introduced a regression
where libvirtd process exits after 120s of
inactivity.
Add a workaround to unset 120s timeout for
libvirtd, the workaround can be removed once
the fix is available in systemd rpm.

Related-Bug: #2029335
Change-Id: Id6db6c17518b54d5fef7c381c509066a569aff6d
2023-08-02 13:13:07 +05:30
Clark Boylan a40f9cb91f Add option to install everything in global venvs
Since we are python3 only for openstack we create a single python3
virtualenv to install all the packages into. This gives us the benefits
of installing into a virtualenv while still ensuring coinstallability.
This is a major change and will likely break many things.

There are several reasons for this. The change that started this effort
was pip stopped uninstalling packages which used distutils to generate
their package installation. Many distro packages do this which meant
that pip installed packages and distro packages could not coexist in the
global install space. More recently git has made pip installing repos as
root more difficult due to file ownership concerns.

Currently the switch to the global venv is optional, but if we go down
this path we should very quickly remove the old global installation
method as it has only caused us problems.

Major hurdles we have to get over are convincing rootwrap to trust
binaries in the virtualenvs (so you'll notice we update rootwrap
configs).

Some distros still have issues, keep them using the old setup for now.

Depends-On: https://review.opendev.org/c/openstack/grenade/+/880266
Co-Authored-By: Dr. Jens Harbott <frickler@offenerstapel.de>
Change-Id: If9bc7ba45522189d03f19b86cb681bb150ee2f25
2023-08-02 07:07:25 +02:00
Dan Smith d115bfd72a Reduce the flush frequency of dbcounter plugin
This relaxes the limits for dbcounter to make it flush stats to the
database less often. Currently every thirty seconds or 100 hits, we
write a stats line to the database. In some services (like keystone)
this can trigger more than one write per second because of the
massive number of SELECT calls that service makes.

This removes the hit limit and decreases the mandatory flush interval
to once a minute. Hopefully this will manifest as lower load on the
database triggered by what would be readonly operations.

Change-Id: I43a58532c0541075a2d36408abc50a41f7994bda
2023-07-31 07:20:12 -07:00
Zuul b314d07e34 Merge "Fix reboot on fedora like nodes" 2023-07-25 16:18:54 +00:00
Zuul c152a409dd Merge "Remove usage of neutron-debug since it has been removed" 2023-06-12 17:20:11 +00:00
Martin Kopec a37b6abc8e Resolve distutils deprecation warning
The distutils package is deprecated and slated for removal in
Python 3.12. Let's use shutil.which which is also recomended
by PEP 632: https://peps.python.org/pep-0632/#migration-advice

Closes-Bug: #2009229
Change-Id: Ibb2a9731449e765c4a56952a9f02679e9618778b
2023-05-26 13:46:42 +02:00
Brian Haley 6764eab264 Remove usage of neutron-debug since it has been removed
The neutron-debug command was deprecated and finally removed,
so tools/ping_neutron.sh can no longer rely on it to create
a probe namespace. Instead, just try and use any namespace
with the network ID in it, since it's either the DHCP (ML2/OVS)
or Metadata (OVN) namespace, which should work just as well.

As this code is rarely (never?) used, this best-effort attempt
is good enough.

Change-Id: I98c992a2a774ef1fb22cee2e90ee342ab2d537ac
Depends-on: https://review.opendev.org/c/openstack/neutron/+/883081
2023-05-12 16:34:08 -04:00
Zuul b33ec4bf1b Merge "Remove support for opensuse" 2023-04-18 17:15:32 +00:00
Sean Mooney 80c3ffe154 Fix reboot on fedora like nodes
This change enables httpd in systemd so that it
starts after a reboot and updates how selinux is
disabled to use /etc/selinux/config in addtion
to setenforce.

Change-Id: I5ea8693c0b967937483bd921b1d9984ea14bc723
2023-03-27 20:58:37 +00:00
Michael Johnson f834f9adaf Fix NotImplementedError in dbcounter on SQLA 2.x
This patch fixes a NotImplementedError raised in the dbcounter plugin when using SQLAlchemy 2.x. The plugin signature has changed and now requires an "update_url" method as part of the plugin[1].
This patch also updates the do_incr() explicit SQL string to use a TextClause and the new requirement for named bound parameters[2].

Closes-Bug: #2009521

[1] https://docs.sqlalchemy.org/en/20/changelog/migration_14.html#changes-to-createengineplugin
[2] https://docs.sqlalchemy.org/en/20/changelog/migration_20.html#execute-method-more-strict-execution-options-are-more-prominent

Change-Id: Ie5484597057a3306757cc46b657446ad61ac2098
2023-03-06 20:09:10 +00:00
Martin Kopec ec07b343d2 Remove support for opensuse
We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.

Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc
2023-02-16 12:01:39 +01:00
Martin Kopec 818d1a225d [doc] Update Ubuntu to 22.04
This updates documentation to reflect the switch to
Ubuntu 22.04 (jammy) in the CI:
https://review.opendev.org/c/openstack/devstack/+/860795

Change-Id: I8bee430029dcc719629bd92451c2791571f8a30c
2022-11-28 11:19:45 +01:00
Rodolfo Alonso Hernandez d1c2bf5e7c Add new service "file_tracker"
This new service periodically tracks the file open in the system.

Closes-Bug: #1995502
Change-Id: I02e097fef07655ff571af9f35bf258b2ed975098
2022-11-07 08:21:34 +00:00
Dan Smith fe7cfa6b8c Avoid including bad service names in perf.json
Some of the API services are not properly mounted under /$service/
in the apache proxy. This patch tries to avoid recording data
for "services" like "v2.0" (in the case of neutron) by only adding
names if they're all letters. A single warning is emitted for any
services excluded by this check.

For the moment this will mean we don't collect data for those services,
but when their devstack API config is fixed, they'll start to show
up.

Change-Id: I41cc300e89a4f97a008a8ba97c91f0980f9b9c3f
2022-06-23 09:25:22 -07:00
Zuul 8eede1fc39 Merge "Add support for IPv6 tunnel endpoints" 2022-06-08 21:39:03 +00:00
Dr. Jens Harbott e6e7100e85 Don't install pinned setuptools with distro pip
We are seeing failures when using an updated setuptools version
installed together with distro pip on Ubuntu 22.04. Install the version
from u-c only when we are also installing pip from upstream.

Change-Id: Ibb6e9424e5794ccbf9a937d2eecfa3bf60ed312e
2022-06-07 13:42:54 +02:00
Brian Haley c869d59857 Add support for IPv6 tunnel endpoints
Currently, neutron tunnel endpoints must be IPv4 addresses,
i.e. $HOST_IP, although IPv6 endpoints are supported by most
drivers.

Create a TUNNEL_IP_VERSION variable to choose which host IP
to use, either HOST_IP or HOST_IPV6, and configure it in the
OVS and Linuxbridge agent driver files. The default is still
IPv4, but it can be over-ridden by specifying TUNNEL_ENDPOINT_IP
accordingly.

This behaves similar to the SERVICE_IP_VERSION option, which
can either be set to 4 or 6, but not 4+6 - the tunnel overhead
should be consistent on all systems in order not to have MTU
issues.

Must set the ML2 overlay_ip_version config option to match
else agent tunnel sync RPC will not work.

Must set the OVN external_ids:ovn-encap-ip config option to
the correct address.

Updated 'devstack-ipv6-only' job definition and verification role
that will set all services and tunnels to use IPv6 addresses.

Closes-bug: #1619476

Change-Id: I6034278dfc17b55d7863bc4db541bbdaa983a686
2022-06-07 02:28:51 +00:00
Zuul 906cf81528 Merge "Add apache2 to the services we collect for memory" 2022-06-03 18:44:10 +00:00
Zuul 0ae279b54a Merge "Fix doc and user create script to set homedir permissions" 2022-05-31 21:15:54 +00:00
Zuul d0657a02e5 Merge "Do not barf stack trace if stats DB is missing" 2022-05-31 19:04:26 +00:00
Dan Smith e85c68e60f Add apache2 to the services we collect for memory
Change-Id: Ic6daef5b4df50ce43c6782542cb54c1958e54655
2022-05-26 09:31:36 -07:00
yatinkarel c64ea4f213 Fix doc and user create script to set homedir permissions
RHEL based distros set homedir permissions to 700,
and Ubuntu 21.04+ to 750[1], i.e missing executable
permission for group or others, this results into failures
as defined in the below bug.

Since in doc we add useradd command, it's good to
add instructions to fix the permissions there itself
instead of getting failures during installation and then
fixing it.

Also update user create script to fix permissions
by adding executable bit to DEST directory if missing.

[1] https://discourse.ubuntu.com/t/private-home-directories-for-ubuntu-21-04-onwards/19533

Closes-Bug: #1966858
Change-Id: Id2787886433281238eb95ee11a75eddeef514293
2022-05-25 06:34:30 +00:00
Zuul 8e1d5aa22e Merge "Drop openEuler support" 2022-05-24 11:34:09 +00:00
Dan Smith 1cdf413ac6 Do not barf stack trace if stats DB is missing
This can happen if devstack fails to run, but we still run the post
tasks. Also could happen if some sort of hybrid job configuration
does not run all of devstack but we still end up running post jobs.

Just warn to stderr and assume no DB info.

Change-Id: I211a331ab668dbb0ad7882908cca4363f865d924
2022-05-23 13:56:13 -07:00
Clark Boylan 50e3c06ec2 Fix dbcounter installation on Jammy
There are two problems with dbcounter installation on Jammy. The first
is straightforward. We have to use `py_modules` instead of `modules` to
specify the source file. I don't know how this works on other distros
but the docs [0] seem to clearly indicate py_modules does this.

The second issue is quite an issue and requires story time. When
pip/setuptools insteall editable installs (as is done for many of the
openstack projects) it creates an easy-install.pth file that tells the
python interpreter to add the source dirs of those repos to the python
path. Normally these paths are appended to your sys.path. Pip's isolated
build env relies on the assumption that these paths are appeneded to the
path when it santizes sys.path to create the isolated environemnt.

However, when SETUPTOOLS_SYS_PATH_TECHNIQUE is set to rewrite the paths
are not appended and are inserted in the middle. This breaks pip's
isolated build env which broke dbcounter installations. We fix this by
not setting SETUPTOOLS_SYS_PATH_TECHNIQUE to rewrite. Upstream indicates
the reason we set this half a decade ago has since been fixed properly.

The reason Jammy and nothing else breaks is that python3.10 is the first
python version to use pip's isolated build envs by default.

I've locally fiddled with a patch to pip [1] to try and fix this
behavior even when rewrite is set. I don't plan to push this upstream
but it helps to illustrate where the problem lies. If someone else would
like to upstream this feel free.

Finally this change makes the jammy platform job voting again and adds
it to the gate to ensure we don't regress again.

[0] https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
[1] https://paste.opendev.org/show/bqVAuhgMtVtfYupZK5J6/

Change-Id: I237f5663b0f8b060f6df130de04e17e2b1695f8a
2022-05-20 10:35:18 -07:00
Dr. Jens Harbott 560ee16a85 Drop openEuler support
The job is broken since it is running with python3.7 and most services
now require at least python3.8.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ie21f71acffabd78c79e2b141951ccf30a5c06445
2022-05-19 14:06:11 +02:00
Dan Smith 64d68679d9 Improve API log parsing
Two runs of the same job on the same patch can yield quite different
numbers for API calls if we just count the raw calls. Many of these
are tempest polling for resources, which on a slow worker can require
many more calls than a fast one.

Tempest seems to not change its User-Agent string, but the client
libraries do. So, if we ignore the regular "python-urllib" agent
calls, we get a much more stable count of service-to-service API
calls in the performance report.

Note that we were also logging in a different (less-rich) format for
the tls-proxy.log file, which hampers our ability to parse that
data in the same format. This switches it to "combined" which is used
by the access.log and contains more useful information, like the
user-agent, among other things.

Change-Id: I8889c2e53f85c41150e1245dcbe2a79bac702aad
2022-05-12 07:55:30 -07:00
Dan Smith fe52d7f0a8 Change DB counting mechanism
The mysql performance_schema method for counting per-database queries
is very heavyweight in that it requires full logging (in a table) of
every query. We do hundreds of thousands in the course of a tempest
run, which ends up creating its own performance problem.

This changes the approach we take, which is to bundle a very tiny
sqlalchemy plugin module which counts just what we care about in
a special database.

It is more complex than just enabling the features in mysql, but it
is a massively smaller runtime overhead. It also provides us the
opportunity to easily zero the counters just before a tempest run.

Change-Id: I361bc30bb970cdaf18b966951f217862d302f0b9
2022-05-12 07:55:02 -07:00
Zuul 85c2999e27 Merge "Tolerate missing deps in get-stats.py" 2022-04-27 22:49:07 +00:00
Dan Smith 1b601c7b1e Tolerate missing deps in get-stats.py
In order to run on systems where not all requirements are present,
we should be tolerant of missing external dependencies, such as
psutil and pymysql. Print a warning (to stderr) and just leave out
those stats in that case.

Also make running the stats collector use ignore_errors:yes to avoid
failures in the future. I think the stats is not critical enough to
fail a job for bugs like this.

Related-Bug: #1970195
Change-Id: I132b0e1f5033c4f109a8b8cc776c0877574c4a49
2022-04-26 08:02:39 -07:00
Harald Jensås bab0c92103 Use tryint() for stats value
In some cases the value is [not set], in this case
the conversion to integer does not work.

Closes-Bug: #1970431
Change-Id: I74df7d8bc9f5cbe0709a6471cf7639caea0b58e8
2022-04-26 15:51:35 +02:00
Dan Smith c2772c2984 Gather performance data after tempest
This makes us gather a bunch of consistent statistics after we run
tempest that can be use to measure the impact of a given change. These
are stable metrics such as "number of DB queries made" and "how much
memory is each service using after a tempest run."

Note that this will always run after devstack to generate the JSON
file, but there are two things that control its completeness:

 - MYSQL_GATHER_PERFORMANCE must be enabled to get per-db stats
 - Unless tls-proxy is enabled, we will only get API stats for keystone

Change-Id: Ie3b1504256dc1c9c6b59634e86fa98494bcb07b1
2022-04-20 13:07:22 -07:00
Rodolfo Alonso Hernandez a756f4b968 Add python3.6 pip support
Since pip v22, python3.6 is not supported (the minimum version is
python3.7). This patch adds the reference for the pip3.6 URL to be
used instead of the default one.

Closes-Bug: #1959600
Change-Id: Iab2c391d5388461fe9e9037cee81884ce8032e72
2022-02-01 11:27:05 +00:00
Dr. Jens Harbott d6909e41af
Use distro pip on Ubuntu
Running get-pip.py fails on Ubuntu when running twice, e.g. after a
unstack/stack cycle. Just use distro pip instead.

Closes-Bug: #1957048
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I87a8d53ed8860dd017a6c826dee6b6f4baef3c96
2022-01-22 17:02:29 +01:00
yatinkarel d5d0bed479 Workaround CentOS 8-stream bug until fixed
Recent iputils release in CentOS 8-stream causing
ping failures with non root user. This needs a fix
in systemd package as mentioned in the Related Bugs,
until it's fixed and is in 8-stream mirrors let's
workaround it by setting net.ipv4.ping_group_range
setting manually.

Related-Bug: #1957941
Related-Bug: rhbz#2037807
Change-Id: I0d8dac910647968b625020c2a94e626ba5255058
2022-01-17 08:43:52 +00:00
Kevin Zhao 7880ba665e openEuler 20.03 LTS SP2 support
openEuler is an open-source Linux based operating system. The current
openEuler kernel is based on Linux and supports multi arch, such as X86_64
and aarch64. It fully unleashes the potential of computing chips. As an
efficient, stable, and secure open-source OS built by global open-source
contributors, openEuler applies to database, big data, cloud computing,
and AI scenarios. openEuler is using RPM for package management.

Note:
Currently there is no available package for uwsgi-plugin-python3 and ovn, so that
openEuler needs manually install them from source.

Website: https://www.openeuler.org/en/

Change-Id: I169a0017998054604a63ac6c177d0f43f8a32ba6
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
2021-12-22 14:47:27 +08:00
Alfredo Moralejo 5ea4c3c18c Support CentOS Stream 9
This patch includes changes required to run devstack on CentOS Stream 9
which has been already published in official repos by CentOS team [1]:

- Add RDO deps repository for CS9.
- remove xinetd package from installation for swift. Note that
  rsync-daemon is installed which should work fine.
- Replace genisoimage by xorriso in CS9.
- Use /etc/os-release to identify the distro in CS9 as it doesn't
  provide lsb_release command.
- Use pip from rpm package instead of from get-pip.py as done in Fedora.
- Add non-voting job devstack-platform-centos-9-stream to the check
  pipeline.

Change-Id: Ic67cddabd5069211dc0611994b8b8360bcd61bef
2021-12-02 09:10:48 +01:00
Zuul fb2e741008 Merge "Stop creating a keystone admin site" 2021-11-03 12:30:07 +00:00
Jens Harbott c2491bac9d Stop creating a keystone admin site
Keystone no longer has any special functionality hidden behind the admin
site. KEYSTONE_AUTH_URI which used to point to the admin site has long
ago been changed to be a copy of KEYSTONE_SERVICE_URI, which points to
the public site.

Drop all KEYSTONE_AUTH_* variables except KEYSTONE_AUTH_URI which may
still be in use in some plugins.

This also allows to finally drop the fixup_keystone() function.

Change-Id: I549f3cadc27d137e014241cdd47e90267859c848
2021-10-17 17:11:03 +02:00
Jens Harbott ee1c614eda Fix use of yaml.load()
The use of this function has been deprecated for a long time[0]. With
PyYAML==6.0 the call is now failing, so replace it with the safe
version.

[0] https://msg.pyyaml.org/load

Signed-off-by: Jens Harbott <frickler@offenerstapel.de>
Change-Id: I7a170262b50a5c80a516095b872d52e1bea5479d
2021-10-16 17:33:12 +02:00
Zuul 82facd6edf Merge "Fix updating setuptools in Centos" 2021-10-12 20:43:00 +00:00
Zuul 10d20b14e7 Merge "Further fixup for Ubuntu cloud images" 2021-10-11 09:26:28 +00:00
Zuul 23cbf138b4 Merge "tools: Fix use of continue" 2021-10-11 08:10:36 +00:00
Zuul e2e88dc19a Merge "Fix displaying usage for make_cert.sh" 2021-10-11 00:53:22 +00:00