We were seeing infinite recursion opening an ssl socket when running
various combinations of python3, eventlet, and urllib3. It is not
clear exactly what combination of versions are affected, but for
background there is an example of this issue documented here:
https://github.com/eventlet/eventlet/issues/371
The immediate cause in nova's case was that we were calling
eventlet.monkey_patch() after importing urllib3. Specifically, change
Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the
nova.utils.monkey_patch() method to make monkey patching common
between WSGI and non-WSGI services. Unfortunately, before executing
this method you must first import nova.utils, which imports a large
number of modules itself. Anything imported (transitively) by
nova.utils would therefore be imported before monkey patching, which
included urllib3. This triggers the infinite recursion problem
described above if you have an affected combination of library
versions.
While this specific issue may eventually be worked around or fixed in
eventlet or urllib3, it remains true that eventlet best practises are
to monkey patch as early as possible, which we were not doing. To
avoid this and hopefully future similar issues, this change ensures
that monkey patching happens as early as possible, and only a minimum
number of modules are imported first.
This change fixes monkey patching for both non-wsgi and wsgi callers:
* Non-WSGI services (nova/cmd)
This is fixed by using the new monkey_patch module, which has minimal
dependencies.
* WSGI services (nova/api/openstack)
This is fixed both by using the new monkey_patch module, and by moving
the patching point up one level so that it is done before importing
anything in nova/api/openstack/__init__.py.
This move causes issues for some external tools which load this path
from nova and now monkey patch where they previously did not. However,
it is unfortunately unavoidable to enable monkey patching for the wsgi
entry point without major restructuring. This change includes a
workaround for sphinx to avoid this issue.
This change has been through several iterations. I started with what
seemed like the simplest and most obvious change, and moved on as I
discovered more interactions which broke. It is clear that eventlet
monkey patching is extremely fragile, especially when done implicitly at
module load time as we do. I would advocate a code restructure to
improve this situation, but I think the time would be better spent
removing the eventlet dependency entirely.
Co-authored-by: Lee Yarwood <lyarwood@redhat.com>
Closes-Bug: #1808975
Closes-Bug: #1808951
Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95
If system time is adjusted first forward and then backward while a
nova service is running (e.g. nova-compute), then there is a high
probability, that periodic tasks will stop for the duration of time
the system clock was adjusted backward.
This was supposed to be fixed by the following patch to oslo.service
https://review.openstack.org/#/c/286838/ , but the order of imports
in unit tests and production code is different, so nova services
end up starting with the default eventlet hub, that does not use a
monotonic clock and, thus, is affected by changes of system time.
Testing this is problematic, as it's a subject of imports order and
is not reproduced in functional or unit tests (oslo_service is always
imported earlier than eventlet hub is initialized, so it just does
"the right thing"). The alternative is to make an assertion when
services start.
Closes-Bug: #1510234
Change-Id: I110cf31ad2a0c74a0cf30ec08bd94d3a56727b39
Instead of working around this in 3 places, only do the import
override at the top level. There were comments left by mikal to do
this after the pbr switch which is now done.
Change-Id: I3ec281dd016ad904e9f323630324e70a002a037d
We don't need to have the vi modelines in each source file,
it can be set in a user's vimrc if required.
Also a check is added to hacking to detect if they are re-added.
Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6
Closes-Bug: #1229324
The gettext.install() function installs a builtin _() function which
translates a string in the translation domain supplied to the install()
function. If gettext.install() is called multiple times, it's the last
call to the function which wins and the last supplied translation domain
which is used e.g.
>>> import os
>>> os.environ['LANG'] = 'ja.UTF-8'
>>> import gettext
>>> gettext.install('keystone', unicode=1, localedir='/opt/stack/keystone/keystone/locale')
>>> print _('Invalid syslog facility')
無効な syslog ファシリティ
>>> gettext.install('nova', unicode=1, localedir='/opt/stack/nova/nova/locale')
>>> print _('Invalid syslog facility')
Invalid syslog facility
Usually this function is called early on in a toplevel script and we
assume that no other code will call it and override the installed _().
However, in Nova, we have taken a shortcut to avoid having to call it
explicitly from each script and instead call it from nova/__init__.py.
This shortcut would be perfectly fine if we were absolutely sure that
nova modules would never be imported from another program. It's probably
quite incorrect for a program to use nova code (indeed, if we wanted to
support this, Nova code shouldn't use the default _() function) but
nevertheless there are some corner cases where it happens. For example,
the keystoneclient auth_token middleware tries to import cfg from
nova.openstack.common and this in turn causes gettext.install('nova')
in other projects like glance or quantum.
To avoid any doubt here, let's just rip out the shortcut and always
call gettext.install() from the top-level script.
Change-Id: If4125d6bcbde63df95de129ac5c83b4a6d6f130a
The moduelauthor tags are used by the docs to see who wrote what.
But many of them are outdated and wrong.
fixes bug 1044578
Change-Id: I9cc27430c906d1418faf28e1b53f6306c62e8f98
The code in nova for logging added an extra root handler to be
able to log messages to the console during log setup. This handler
was removed in the setup method. The common setup method no longer
removes this handler, so don't create it. Note that there may be
a small period before the logging setup is finished where messages
will not appear.
Fixes bug 1053512
Change-Id: I879360ecd60d607112b1210d7afd5860e9ccb295
* Fixes lp904305
* remove module level log functions (getLogger to rule them all)
* Move specific Environment logging to the one place it is used
* Wrap getLogger to return a logger wrapped in a NovaContextAdapter
* Do not overwrite the root logger
* save_and_reraise_exception logs via error for passing exc_info
* Uses CommonConfigOptions for compatability across Openstack Projects
* Prefers CommonConfigOptions over legacy options
* Install a NullHandler on the root logger if configured by FLAGS
* Include example logging config file to mimic Nova defaults
Change-Id: Ie59c3f755c142e2b7dc3b94b4e82e142e157bfac
part 1 of blueprint separate-nova-metadata
* adds api/metadata/ and moves code from ec2
* moves metadata into separate binary
* changes metadata forward to use metadata host and port
* moves the metadata accept rule to the metadata api
* adds nova-api-* to setup.py
Change-Id: I7f5d8e6cafc55b5c383cd88991f29c6059fb8d82
Matthew Booth