openstack
/
nova
Code Issues Proposed changes

Add reno about nova-api eventlet monkey-patching and rabbitmq 

This adds a known issue release note about eventlet monkey-patching in
nova-api and workarounds.

Related-Bug: #1825584
Related-Bug: #1829062

Change-Id: I22abd1f5377489dd809eb705c8e7aec2814ced0e
This commit is contained in:
melanie witt 2019-05-29 21:32:11 +00:00 committed by Matt Riedemann
parent 498608c15f
commit a694952eac
1 changed files with 42 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
releasenotes/notes/eventlet-monkey-patch-5f734ef581aa550e.yaml Normal file
View File

@ -0,0 +1,42 @@
---
issues:
- |
Operators should be aware that nova-api has a dependency on eventlet for
executing parallel queries across multiple cells and is monkey-patched
accordingly. When nova-api is running under uWSGI or mod_wsgi, the wsgi
app will pause after idle time. While the wsgi app is paused, rabbitmq
heartbeats will not be sent and log messages related to this can be seen
in the nova-api logs when the wsgi app resumes when new requests arrive to
the nova-api. These messages are not harmful. When the wsgi app resumes,
oslo.messaging will reconnect to rabbitmq and requests will be served
successfully.
There is one caveat, which is that the wsgi app configuration must be left
as the default ``threads=1`` or set explicitly to ``threads=1`` to ensure
that reconnection will work properly. When threads > 1, it is not
guaranteed that oslo.messaging will reconnect to rabbitmq when the wsgi app
resumes after pausing during idle time. Threads are used internally by
oslo.messaging for heartbeats and more, and it may fail in a variety of
ways if run under eventlet with an app that violates eventlet's threading
guarantees. When oslo.messaging does not reconnect to rabbitmq after a
wsgi app pause, RPC requests will fail with a ``MessagingTimeout`` error.
So, it is necessary to have the wsgi app configured with ``threads=1`` for
reconnection to work properly.
If running with ``threads=1`` is not an option in a particular environment,
there are two other workarounds:
* Use the eventlet wsgi server instead of uWSGI or mod_wsgi, or
* Disable eventlet monkey-patching using the environment variable
``OS_NOVA_DISABLE_EVENTLET_PATCHING=yes``.
Note that disabling eventlet monkey-patching will cause queries across
multiple cells to be serialized instead of running in parallel and this
may be undesirable in a large deployment with multiple cells, for
performance reasons.
Please see the following related bugs for more details:
* https://bugs.launchpad.net/nova/+bug/1825584
* https://bugs.launchpad.net/nova/+bug/1829062