pbr's 'wsgi_scripts' entrypoint functionality is not long for this world
so we need to start working towards an alternative. We could start
packaging our own WSGI scripts in DevStack but using module paths seems
like a better option, particularly when it's supported by other WSGI
servers like gunicorn.
Currently only nova is migrated. We should switch additional projects as
they migrate and eventually remove the support for WSGI scripts
entirely.
Change-Id: I057dc635c01e54740ee04dfe7b39ef83db5dc180
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Depends-on: https://review.opendev.org/c/openstack/nova/+/902687/
This will cause apache to no longer wait forever for a connection
pool member to become available before returning 503 to the client.
This may help us determine if some of the timeouts we see when
talking to the services come from an overloaded apache.
Change-Id: Ibc19fc9a53e2330f9aca45f5a10a59c576cb22e6
This change enables httpd in systemd so that it
starts after a reboot and updates how selinux is
disabled to use /etc/selinux/config in addtion
to setenforce.
Change-Id: I5ea8693c0b967937483bd921b1d9984ea14bc723
We haven't been testing the distro for a while in CI, e.g. in
Tempest, the jobs on opensuse15 haven't been executed for a year
now.
Therefore the patch removes opensuse support from devstack.
Closes-Bug: #2002900
Change-Id: I0f5e4c644e2d14d1b8bb5bc0096d1469febe5fcc
openEuler 20.03 LTS SP2 support was removed from devstack in last
few months due to its python version is too old and the CI job
always fail. And openEuler 20.03 LTS SP2 was out of maintainer in May
2022 by openEuler community.
The newest LTS version was released in March 2022 called 22.03 LTS.
This release will be maintained for at least 2 years. And the python
version is 3.9 which works well for devstack.
This Patch add the openEuler distro support back. And add the related
CI job to make sure its works well.
Change-Id: I99c99d08b4a44d3dc644bd2e56b5ae7f7ee44210
The job is broken since it is running with python3.7 and most services
now require at least python3.8.
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ie21f71acffabd78c79e2b141951ccf30a5c06445
There are a couple of places that still use a hardcoded
127.0.0.1 value, even if devstack is run with
SERVICE_IP_VERSION=6 in local.conf. While things still
work, SERVICE_LOCAL_HOST should be used instead since
everything else could be using IPv6.
Change-Id: I2dd9247a4ac19f565d4d5ecb2e1490501fda8bca
openEuler is an open-source Linux based operating system. The current
openEuler kernel is based on Linux and supports multi arch, such as X86_64
and aarch64. It fully unleashes the potential of computing chips. As an
efficient, stable, and secure open-source OS built by global open-source
contributors, openEuler applies to database, big data, cloud computing,
and AI scenarios. openEuler is using RPM for package management.
Note:
Currently there is no available package for uwsgi-plugin-python3 and ovn, so that
openEuler needs manually install them from source.
Website: https://www.openeuler.org/en/
Change-Id: I169a0017998054604a63ac6c177d0f43f8a32ba6
Co-Authored-By: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Kevin Zhao <kevin.zhao@linaro.org>
Since victoria cycle, we have moved upstream testing to
Ubuntu Focal (20.04) and so does no Bionic distro in
Xena cycle testing runtime[1]. Grenade jobs also started
running on Focal since victoria was released.
Only thing left was legacy jobs which were not migrated to
Ubuntu Focal in Victoria and as per another community-wide
goal[2], all the lgeacy jobs were suppsoed to be migrated
to zuulv3 native jobs in victoria cycle itself. One of the
pending job was in nova (nova-grenade-multinode) which is also
migrated to zuulv3 native now
- https://review.opendev.org/c/openstack/nova/+/778885
If there is any job running on bionic, we strongly recommend
to migrate it to Ubuntu Focal.
[1] https://governance.openstack.org/tc/reference/runtimes/xena.html
[2] https://governance.openstack.org/tc/goals/selected/victoria/native-zuulv3-jobs.html
Change-Id: I39e38e4a6c2e52dd3822c9fdea354258359a9f53
Full Glance functionality requires Glance being run in a configuration
where it can spawn long-running task threads. The default uwsgi mode
does not allow this, and the current workaround is to set WSGI_MODE
to something other than uwsgi to get the devstack code to deploy
Glance as a standalone service. Since this affects the entire rest of
the deployment, this patch separates out a flag to control this behavior
specifically for Glance. When WSGI_MODE=uwsgi, control of the Glance
deployment mechanism is allowed via GLANCE_STANDALONE=True|False. If
WSGI_MODE!= uwsgi then we deploy standalone Glance anyway.
Change-Id: I79068ce0bd7414bc48ff534ee22f0de5d7b091cb
This fixes a stack.sh execution error on CentOS8. We should use
python3-mod_wsgi instead of mod_wsgi since mod_wsgi is replaced by
python3-mod_wsgi. The following change may affect this issue.
4a746b53e9?branch=c8s
Change-Id: I5344ecf519e1a79091b6158c2d711d09b21fae0c
Closes-Bug: #1885645
- Add a nodeset and a platform job
- Drop uwsgi-py2 pkg that no longer exists
- Blacklist tests that are currently failing
Change-Id: Ib4416dc2f5e003fd770f5240a8f78213c56af8e6
Building uwsgi from source was a workaround that was introduced a long
time ago, it doesn't seem like it is needed anymore and will actually
fail for Ubuntu 20.04.
Also it doesn't match what will happen for most real-world
installations, so let's try to get back to using distro packages. We'll
still use the source install for RHEL/Centos, it remains to be tested
whether we can get back to using distro versions there, too.
Change-Id: I82f539bfa533349293dd5a8ce309c9cc0ffb0393
Removing the (f23,)f24 support they are EOL.
The only non-trivial change is the apache-httpd default worker change,
however might not be bad idea to use `event` instead of `worker`
in the future, but for now keep it AS-IS and continue to use `worker`.
Change-Id: I96d414a30b58bc4b43da45066fdf310a6a830079
Closes-Bug: #1740194
uwsgi services:
[1] By default uwsgi is set to exit on reload this breaks config reloading
of the service [1][2]. It needs to be set to 'false'.
[2] Requires to add 'systemctl reload' command support by adding ExecReload
in unit file.
Non uwsgi services:
[1] Non uwsgi services only requires to add ExecReload in unit file.
There was a similar patch submitted by Matthew Treinish [3] but it was
already set to workflow +1(not merged as having dependency on other patch)
and it was having some issues as specified in comment and missing reload
functionality for other services.
[1] https://etherpad.openstack.org/p/uwsgi-issues
[2] http://uwsgi-docs.readthedocs.io/en/latest/Options.html#exit-on-reload
[3] https://review.openstack.org/#/c/490904/2
Change-Id: I78f5e9d4574671c74a52af64724946feb41c2d7a
1] Process using uwsgi:
uwsgi services doesn't support for graceful shutting down [1].
It requires some changes in unit files [2] including adding below
graceful shutdown hook and changing KillSignal:
--hook-master-start "unix_signal:15 gracefully_kill_them_all
All the steps and changes required are specified in etherpad [1].
2] Non uwsgi services needs below changes:
In [service] section:
a. Add KillMode = process
b. Add TimeoutStopSec = infinity
NOTE:
Creating unit file for services other than uwsgi is handled by the
'write_user_unit_file' function [3]. This function is common for all
the services so this patch adds the above mentioned parameters for
services using ServiceLauncher also though they don't require.
Added a new stackrc variable WORKER_TIMEOUT which is required to add
graceful shutdown support to uwsgi services. It will be set as a value
to 'worker-reload-mercy' [4] in uwsgi file of service. The default
value set to this variable is 90.
[1] https://etherpad.openstack.org/p/uwsgi-issues
[2] https://www.freedesktop.org/software/systemd/man/systemd.kill.html
[3] 2967ca3dfd/functions-common (L1439-L1461)
[4] http://uwsgi-docs.readthedocs.io/en/latest/Options.html#worker-reload-mercy
Co-Authored-By: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: Ia95291325ce4858b47102dd49504250183f339ab
Calling setenv appears to be globally scoped which is breaking the
glance path which relies on chunked uploads. The glance path is
separated by using mod_proxy instead of mod_proxy_uwsgi because
mod_proxy_uwsgi doesn't support chunked encoding.[1] The proxy-sendcl [2]
was set on the mod_proxy_uwsgi path just in case someone tried to send a
chunked request to the api server we would be able to handle it. It
tells apache to locally cache the chunked request and send the
content-length as a normal upload to the upstream server. However, if we
can only set it globally across then small potential benefit is not worth
having all glance uploads cached by apache. This commit just removes
setting the flag. In the future if we can have devstack isolate this
flag it might be worth adding back to the mod_proxy_uwsgi path, but for
right now it's not worth the tradeoff.
[1] https://github.com/unbit/uwsgi/issues/1540
[2] https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#request-bodies
Depends-On: Idf6b4b891ba31cccbeb53d373b40fce5380cea64
Change-Id: Iab2e2848877fa1497008d18c05b0154892941589
Closes-Bug: #1709970
In trying to debug periodic gate instability of CentOS, I noticed that
it is using the prefork mpm, while Ubuntu is defaulting to the
multi-threaded worker mpm.
One of the problems seems related to 502 proxy errors from the TLS
proxy. We see out-of-sync timestamps in the centos TLS proxy access
logs, which might be innocent behaviour based on the prefork model or
indicate something else.
Before going too deep down this rabbit-hole, I think it is better for
consistency to use the same mpm model on all our platforms, and start
debugging from there.
Change-Id: I9881f2e7d51fdd9fc0f7fb3e37179aa53171b531
This commit increases the socket timeout value from 4 secs to a much
higher 30 secs. This is just for sanity, the load is high when we're
seeing the wsgi.input timeouts, so uwsgi might be just closing the
socket waiting for data over the wire. 30 seconds is overly conservative
just so we can rule this out. This will likely be shrunk to a more
reasonable value in the future.
Change-Id: Iae85d3a084fb33b2a63550d6e353413e98c0b39c
Partial-Bug: #1701088
Previously the local uwsgi server mode was using uwsgi in http mode.
This was unessecary and actually not recommend by the uwsgi docs [1][2]
This is because http mode starts a frontend http process that forwards
requests to the workers running the python code. This is done for the
largely the same reasons we're using apache as a proxy and is
unnecessary. http-socket mode doesn't do this and just exposes the
workers as an http interface to the proxy. (in our case apache)
[1] http://uwsgi-docs.readthedocs.io/en/latest/HTTP.html#http-sockets
[2] http://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html
Change-Id: I5671687c8083fa4bdee066c07b083a0f00be532b
According to the uwsgi docs [1] for http keepalive there is a separate
option for http keep alive, and just setting connection close isn't
enough. This commit makes sure we disable http keepalive. This will
hopefully fix the random connection issues we get on image uploads to
glance, which uses uwsgi http mode.
[1] http://uwsgi-docs.readthedocs.io/en/latest/HTTP.html#http-keep-alive
Change-Id: Ic5f83c5c93f28b2bd62ca9ac96ca8c87797ea5c9
Closes-Bug: #1701088
Unless NOVA_USE_MOD_WSGI is False, run nova-api and nova-metadata
using uwsgi.
Because the metadata server is always expected to run on a port and
without a prefix, we have it configured to use uwsgi but not to
proxy from apache: uwsgi listens on the configured port itself.
uwsgi process that listen themselve do not need a socket or to
chmod-socket, so those config lines have been moved to the block
that is also writing proxy configuration for apache.
Because this change only uses uwsgi for nova-api and nova-api-meta,
nova-api-meta is set to default to enabled in stackrc because the
nova-api wsgi application used by wsgi only presents the one service
(osapi_compute).
If NOVA_USE_MOD_WSGI is False and tls_proxy service is enabled,
nova-api is run on an internal port reached via the tls_proxy.
Depends-On: I8ff08d61520ccf04e32dcd02f4cecc39dae823cb
Change-Id: If2d7e363a6541854f2e30c03171bef7a41aff745
This adds packages to suse for systemd python linkages as well as
apache2 and which. And configures mod_proxy and mod_proxy_uwsgi with
a2enmod.
We also properly query if apache mods are enabled to avoid running
into systemd service restart limits. Enable mod_version across the board
as we use it and it may not be enabled by default (like in SUSE).
Also in addition to enabling mod_ssl we enable the SSL flag so that TLS
will work...
Finally we tell the system to trust the devstack CA.
Change-Id: I3442cebfb2e7c2550733eb95a12fab42e1229ce7
This commit adds support for deploying glance as a wsgi script under
uwsgi. To get around limitations in the uwsgi protocol when using
python3 for chunked encoding we have to setup uwsgi in http mode on a
random port listening on localhost and use mod_proxy to forward the
incoming requests. The alternative approach of having apache buffer the
requests locally with the send_cl option with mod_proxy_uwsgi only
worked on python2 and also has the limitation that apache is buffering
the entire chunked object, which could be several gigabytes in size.
Depends-On: I089a22a4be4227a551c32442dba27c426f54c87d
Change-Id: Ie98fb7da5e8ecfa49cd680b88139cb7034d5f88f
On ubuntu contents of /var/run do not persist between reboots. Devstack
uses /var/run/uwsgi as home for wsgi sockets. This means that after
rebooting the machine services, that rely on uwsgi would fail to start.
Currently it affects keystone.service and placement-api.service.
This patch changes delegates directory creation to systemd-tmpfiles,
which would run on startup.
Change-Id: I27d168cea93698739ef08ac76c828695a49176c7
Closes-Bug: #1692767
As described in [1], it seems that mod_wsgi is not "graceful" reload
safe. Upon re-init, it can end up in a segfault loop.
The "reload" (not *restart*) after setting up uwsgi was added with
I1d89be1f1b36f26eaf543b99bde6fdc5701474fe but not causing an issue
until uwsgi was enabled.
We do not notice in the gate, because the TLS setup ends up doing a
restart after this setup. In the period between the
write_uwsgi_config and that restart, Apache is sitting in a segfault
loop, but we never noticed because we don't try talking to it. Other
jobs that don't do any further apache configuration have started
failing, however.
Looking at the original comments around "reload_apache_server" I'm not
sure if it is still necessary. [2] shows it is not used outside these
two calls.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1445540
[2] http://codesearch.openstack.org/?q=reload_apache_server&i=nope&files=&repos=
Closes-Bug: #1686210
Change-Id: I5234bae0595efdcd30305a32bf9c121072a3625e
On Centos, apache has a private view of /tmp and thus can't see this
socket, causing keystone to fail. This happened after
I46294fb24e3c23fa19fcfd7d6c9ee8a932354702.
Move it to /var/run.
Closes-Bug: #1684360
Change-Id: I47f091656802719c259752454ec88bf50760b967
a2dissite will return a non-zero error code if the site that is being
disabled is not currently enabled (that is, if the conf file for it does
not exist). This can happen during development if you've been messing
with files by hand. Rather than exploding out of a ./stack.sh, accept
the missing file as meaning "it's disabled" and carry one. The rpm
version of disable, which does not use a2dissite, does this already.
Change-Id: Ie5dfd42efdff4bdba5ffaa765af000dd8e1d596e
This makes keystone use the proxy uwsgi module when running in uwsgi
mode. It also introduces a new stackrc variable which is WSGI_MODE
that we can use to control the conditionals in services that current
work with mod_wsgi.
Also update retry timeouts on proxy pass so that workers don't disable
their connections during polling for initial activity.
Change-Id: I46294fb24e3c23fa19fcfd7d6c9ee8a932354702
The uwsgi proxy version that comes with Ubuntu xenial is too old, so
we have to build it from source. This is a temporary solution until
the next LTS.
This lays the ground work for using it in keystone.
Change-Id: I00fb1759e6988c7df0ce0f3df5ff1ce9fd7cd381
Instead of this code all existing in keystone inline, factor out into
a dedicated set of functions, and make keystone use this. This drops
uwsgi supporting https directly, but that's not going to be a
supported model going forward once we get to proxy only anyway.
Change-Id: I1d89be1f1b36f26eaf543b99bde6fdc5701474fe
We're now in a systemd world where systemd is managing the restart
effectively, there is no reason to be tricksy with apache now that
we're not working around weird upstartd issues.
Change-Id: Ifadfd504eb10a90db5177ea9180b9cd8331a2948