We want this module for use elsewhere. Given there's only a single
caller (nova.service) we can simply move the code to the caller.
Change-Id: I2c3887db8b3f6833bf24f5114fd955e1af590d03
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Once again, do what we did for '_LE' and '_LW' and remove the final
remnants of the log translation effort.
Change-Id: Id6cf7a9bfbe69d6d3e65303e62403d1db9188a84
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
We've been slowly removing these as we go. Remove the final few '_LE'
occurrences now.
Change-Id: I75ebd2e95a0c77585d7b4329ca01e4bacc1dd7c4
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
These particular imports are no longer needed in a Python 3-only world.
Change-Id: Ia1b60ce238713b86f126e2d404199d102fdbc5bc
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
The file nova/api/openstack/__init__.py had imported a lot of
modules, notably nova.utils. This means that any code which
runs within that package, notably the placement service, imports
all those modules, even if it is not going to use them. This
results in scripts/binaries that are heavier than they need
to be and in some cases including modules, like eventlet, that
it would feel safe to not have in the stack.
Unfortunately we cannot sinply rename nova/api/openstack/__init__.py
to another name because it contains FaultWrapper and FaultWrapper
is referred to, by package path, from the paste.ini file and that
file is out there in config land, and something we prefer not to
change. Therefore alternate methods of cleaning up were explored
and this has led to some useful changes:
Fault wrapper is the only consumer of walk_class_hierarchy so
there is no reason for it it to be in nova.utils.
nova.wsgi contains a mismash of WSGI middleware and applications,
which need only a small number of imports, and Server classes
which are more complex and not required by the WSGI wares.
Therefore nova.wsgi was split into nova.wsgi and nova.api.wsgi.
The name choices may not be ideal, but they were chosen to limit
the cascades of changes that are needed across code and tests.
Where utils.utf8 was used it has been replaced with the similar (but not
exactly equivalient) method from oslo_utils.encodeutils.
Change-Id: I297f30aa6eb01fe3b53fd8c9b7853949be31156d
Partial-Bug: #1743120
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can
use dict.items instead, as it will return
iterators in PY3 as well. And dict.items/keys
will more readable. 2.In py2, the performance
about list should be negligible, see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html
The patch list:
1. cells.
2. compute api.
3. image.
4. network.
5. objects.
6. scheduler.
7. virt.
8. other resources.
Partial-Implements: blueprint replace-iteritems-with-items
Change-Id: Ic6e469eb80ee1774de1374bb36f38b5134b6b311
Fixes unit tests related to incomparable types.
Fixes unit tests related to writing / reading bytes in / from files
or response bodies.
Fixes console websocketproxy unit tests.
Fixes consoleauth unit tests.
Fixes db unit tests.
Fixes image unit tests.
Fixes virt/block_devices unit tests.
Fixes virt/configdrive unit tests.
Fixes virt_drivers unit tests.
Fixes xenapi agent, vmops and volume_utils unit tests.
Fixes wsgi unit tests.
Enables some python3 unit tests.
Partially Implements: blueprint goal-python35
Change-Id: Ie98e968740d1015eae3278edeb93d4ba08155169
Now that the WSGI opts have been added to a group, there's no longer
any reason to keep the 'wsgi_' prefix for some of these opts around.
Remove it, ensuring the old names are stored as deprecated names.
Change-Id: If39bed3ed2335b6f96455870f1a47cb868d21f39
Move all wsgi configuration into [wsgi] group. Update references to
these configuration options to reflect new group.
Change-Id: I1a0a0ce1b0c1bdf60114caf6f5d4e28fd2c3cda5
The config options of the "nova.conf" section "wsgi"
got moved to the new central location "nova/conf/wsgi.py".
Change-Id: I4bf8ea48c374d5d35fbbd37c125607e05d3679e1
Implements: blueprint centralize-config-options
The error message when providing a backlog argument either zeroed or negative
needs to be i18n and also fixed to match with the conditional.
Change-Id: Ib9e15091826e8a68289d90b53116837770e826c7
Closes-Bug: #1487350
Change Id52c30bb5ded2184d772e6026b0f04f9a0efeb55 added a hacking check
for greenthread.spawn(). Since eventlet.spawn() calls
greenthread.spawn() under the covers, it should also be checked. Because
there are still occurrences of eventlet.spawn(), these were also cleaned
up in order to pass the added hacking check.
Co-Authored-By: Qin Zhao <chaochin@gmail.com>
Change-Id: Ia125b4ad5e84bf48091af5a7a483b89629f0ec31
Related-Bug: #1404268
Closes-Bug: #1468513
Return correct scheme in version URLs if service
behind an SSL termination proxy.
This is done by adding a new configuration option,
secure_proxy_ssl_header, which, when defined, makes
the wsgi application take the host_url scheme from
that header. By default, when this option is not
specified, there is no difference in behavior.
The intention is to configure any ssl-decrypting
proxy to set that header, so that nova-api knows
which protocol to use in the URLs in response.
This patch is largely based on
https://review.openstack.org/#/c/132235/18
DocImpact
Closes-Bug: #1384379
Change-Id: I27ba166902ecc19c9b7fff2ee7f3bf733885efe1
Modules eventlet_backdoor, loopingcall, periodic_task,
service, sslutils, systemd, threadgroup were removed
from nova. These modules were imported from oslo.service
library.
Co-Authored-By: Marian Horban <mhorban@mirantis.com>
Depends-On: I305cf53bad6213c151395e93d656b53a8a28e1db
Change-Id: Iaef67e16af3d69f845742f7bdcb43667bf1576ee
With Eventlet 0.17.2, we can directly pass the logger
instance. This allows us to deprecate/remove the WritableLogger
class from oslo.log.
Closes-Bug: #1440773
Change-Id: I3985b30b499a1e676d39c2a6cd3a31ac13c0ab47
This patch was generated by the sixer tool version 0.2 using the
"iteritems" operation:
https://pypi.python.org/pypi/sixer
Manual changes:
- Don't change get_instance_metadata() in nova/compute/api.py:
fixed by the change Ifd455e70002eb9636b87f83788384127ba6edeeb.
- Don't change sqlalchemy code and
nova/tests/unit/db/test_db_api.py. sqlalchemy objects cannot be
converted to a dictionary using dict(obj) directly yet. It will be
possible with the change I702be362a58155a28482e733e60539d36c039509.
- Revert change in a comment in nova/objects/instance.py; the sixer tool
is limited and don't understand comments
- Reformat nova/virt/vmwareapi/driver.py to respect the 80 columns
contraint
Blueprint nova-python3
Change-Id: I81465661cb8a74778d70ba9b6641073f1effa49b
This creates a noauth2 auth_strategy which is similar to noauth,
except it only gives you an admin context if the username passed in is
'admin'. This allows testing of non admin activities.
noauth is deprecated as of this commit. While we expect that it would
only be used in testing, it is exposed as a conf option, so could be
used behind a different auth proxy.
Also make the error path for pipeline loading contain a full
LOG.exception. This is a fatal condition for nova, and the current
error was often quite opaque. The full stack trace during this fatal
error makes addressing paste.ini issues much more straight forward.
DocImpact
Change-Id: I7cb5ab3e43a1e3bd7ccba0480053361743f859b2
Convert the use of the incubated version of the log module
to the new oslo.log library.
Sync oslo-incubator modules to update their imports as well.
Co-Authored-By: Doug Hellmann <doug@doughellmann.com>
Change-Id: Ic4932e3f58191869c30bd07a010a6e9fdcb2a12c
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware
Change-Id: I3e2eb147b321ce3e928817b62abcb7d023c5f13f
Currently socket options, socket.SO_REUSEADDR, socket.TCP_KEEPIDLE and
socket.SO_KEEPALIVE are set only if SSL is enabled.
socket.SO_REUSEADDR:
This socket option tells the kernel that even if this port is busy
(in the TIME_WAIT state), go ahead and reuse it anyway. It is useful if your
server has been shut down, and then restarted right away while sockets are
still active on its port.
socket.SO_KEEPALIVE:
To confirm that an idle connection is still active, these implementations
send a probe segment designed to elicit a response from the peer TCP. The the
probe causes the receiver to return an acknowledgement segment, confirming
that the connection is still live. If the peer has dropped the connection due
to a network partition or a crash, it will respond with a RST instead of an
acknowledgement segment.
socket.TCP_KEEPIDLE:
It is useful to set this socket option, because if the other peers lose their
connection (for example by rebooting) you will notice that the connection is
broken, even if you don't have traffic on it. If the probes are not replied
to by your peer, you can assert that the connection cannot be considered valid
and then take the correct action.
IMO, these options are useful in both the cases, whether SSL is enabled or not.
Made provision to set socket.SO_REUSEADDR, socket.TCP_KEEPIDLE and
socket.SO_KEEPALIVE socket options if SSL is enabled or not.
Closes-Bug: #1369414
Change-Id: I25b353dcf1ca6eba1c54d297994d56c0064daca5
Presently, the wsgi server allows persist connections hence even after
the response is sent to the client, it doesn't close the client socket
connection.
Because of this problem, the green thread is not released back to the pool.
In order to close the client socket connection explicitly after the
response is sent and read successfully by the client, you simply have to
set keepalive to False when you create a wsgi server.
Add a parameter to take advantage of the new(ish) eventlet socket timeout
behaviour. Allows closing idle client connections after a period of
time, eg:
$ time nc localhost 8776
real 1m0.063s
Setting 'client_socket_timeout = 0' means do not timeout.
DocImpact:
Added wsgi_keep_alive option (default=True).
Added client_socket_timeout option (default=900).
SecurityImpact
Closes-Bug: #1361360
Change-Id: I399b812f6d452226fd306c423de8dcea8520d2aa
oslo.i18n uses different marker functions to separate the
translatable messages into different catalogs, which the translation
teams can prioritize translating. For details, please refer to:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html#guidelines-for-use-in-openstack
There were not marker fuctions some places in directory network.
This commit makes changes:
* Add missing marker functions
* Use ',' instead of '%' while adding variables to log messages
Change-Id: Iaebb239ef20a0da3df1e3552baf26f412d0fcdc0
oslo.i18n uses different marker functions to separate the
translatable messages into different catalogs, which the translation
teams can prioritize translating. For details, please refer to:
http://docs.openstack.org/developer/oslo.i18n/guidelines.html#guidelines-for-use-in-openstack
There were not marker fuctions some places in directory network.
This commit makes changes:
* Add missing marker functions
* Use ',' instead of '%' while adding variables to log messages
Change-Id: I5a8f381b6f8fdb4e8febe9e6a901f7cdc6846646
oslo.utils library now provides the functionality previously in
oslo-incubator's excutils, importutils, network_utils, strutils
timeutils, units etc. Some modules already moved to oslo.utils
will still be around since other code in nova/openstack/common/
are using it and will be removed in a subsequent commit.
Change-Id: Idc716342535fdfa680963e0e073ddb46f5f1eb34
This is the same as using locals() for variable replacement in a log
message which violates hacking rule H501. On Cinder this actually
caused the cinder-api process to hang on python 2.6.
Nova has the same code so this fixes the hacking violation.
Note that the H501 rule was updated in commit 55e96ee but it's not in a
released version of hacking yet (and won't be in time for Juno).
Closes-Bug: #1365901
Change-Id: I1be0db327a2fa1e866328eecd2e7c3dd3ee3cc93
oslo.i18n provides the i18n functions that were provided by
oslo-incubator's gettextutils module. Some tests that were
using internal details of the library were removed.
Change-Id: I44cfd5552e0dd86af21073419d31622f5fdb28e0
Both H305 and H307 are part of the OpenStack style guide.
Ensure that there are 3 groups of imports, stdlib,
third-party and project specific. Within each category
ensure that imports are in alphabetical order.
Change-Id: Id4994551c135c3c1a4982153f0c6cacba6176b95
Added reset method in WSGIService class.
After adding reset method when SIGHUP signal is sent to
wsgi service parent process,then it sends SIGHUP signal
to all of its child processes. Each child process handles
SIGHUP signal by first stopping the service and then calls
service start method again. When it stops the service, it
kills the eventlet thread, which internally closes the wsgi
server socket object. This server socket object is now not
usable again and it throws following error, while restarting
the service:
error: [Errno 9] Bad file descriptor
To resolve 'Bad file descriptor' error, creating duplicate
socket object, every time service starts.
When the wsgi service is stopped, it sets the green pool
size to 0, so resizing the green pool to default value,
on service restart.
Closes-Bug: #1334651
Change-Id: If1035deaf8b31f2712d88f0112fdd2e9e9dc7cb0
When SIGHUP signal is sent to nova-api service, it doesn't complete
processing of all pending requests before terminating all the
processes.
Closes-Bug: #1334647
Change-Id: I3f7bd0ba5ef1bb716a97635252abe251053a669d
Our translation policy
(https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation) calls
for not translating debug level logs. This is to help prioritize log
translation. Furthermore translation has a performance overhead, even if
the log isn't used (since nova doesn't support lazy translation yet).
This is the final patch ensuring that all fines in the nova directory
no longer have translations for debug messages.
Change-Id: Ib10f5826e2238cbb2d4315c48be570eb31b2ea55
Currently HTTPForbidden is using wrong parameter 'detail'
somewhere, while the correct parameter should be 'explanation'.
This patch fixes this bug.
Change-Id: I69bc3249f88e9cbf9add954571c21b07cf58f7c6
Closes-Bug: #1283872
HTTP messages max header line size has been increased from 8K to 16K
by default to allow using tokens including up to 14-15 catalog
entries. The same issue (https://bugs.launchpad.net/swift/+bug/119014)
may occur again in the future, if Keystone's catalog grows further.
Allowing operators to customize the max header size, will allow them
to have Nova working whatever the size of the catalog (if the option
is properly set).
Related-Bug: #1190149
DocImpact
Change-Id: I96694ac9bc242267cfc9f4d0c18b2b7fe0000460
* nova/wsgi.py(Service.__init__): Catch external errors
(socket.error is a subclass of EnvironmentError since python 2.6),
and log the port number attempted to bind to.
Closes-Bug: 1237961
Change-Id: Ia4eb46c76db855fc3168c5b7a233ce94d2ef7726
We don't need to have the vi modelines in each source file,
it can be set in a user's vimrc if required.
Also a check is added to hacking to detect if they are re-added.
Change-Id: I347307a5145b2760c69085b6ca850d6a9137ffc6
Closes-Bug: #1229324
the debug flag basically just means that we get lines like:
....wsgi.server [-] (6966) accepted ('127.0.0.1', 51944)
After *every single* wsgi request, filling up the logs with
completely irrelevant data at info level because eventlet.wsgi
does not use python logging.
Change-Id: I508b963a0c84e17df845aad260dd8ee157abbea8
If api_paste_conf is specified as a relative path and the directory from
where the nova-api service is started contains a file with the same
name, the wsgi loader won't try to find the absolute path to that file
which is required when not passing the base directory to the
paste.deploy library.
This change ensures the config_path is an absolute path so it can be
found by the paste.deploy library.
Closes-Bug: #1259183
Change-Id: I0eb6ef1d69d302634b8c842449f997ec10c13933
Expose the number of greenthreads in wsgi pool to configuration. This
allows to put a limit to the number of resources like memcache
connections used.
DocImpact
Change-Id: I6e53d6e12661ed2f42a6b543891b13eec0c8300c
The following commit brings in oslo-incubator service.py which
includes calls to support graceful shutdown. The common service
class has changed to handle signals and manage Services. When
a signal is received the Services are stopped which in turn
waits on all child threads (Proxy callbacks) to complete.
The oslo-incubator commit hash that was used to update
nova was b5fba9e40d609d63b4dab9ca351f7e22e9d8b35b.
This allows long running commands to complete prior to killing
the process. This change only works for SIGTERM and SIGINT
signals.
Unit test was added to verify parent stop method is called in
a shutdown condition.
Change-Id: I21a45862f491530523037632fd70c60cc0153c0a
Implements: blueprint graceful-shutdown
This is for Python 3 compatibility, since Python 3 does not support
the print statement, only the print function. (Python 2.6 and 2.7
support both, depending on whether a __future__ import is used.)
Paths containing "xen" were left alone, because of comments in a
couple of the files indicating that the XenServer code needed to be
compatible with Python 2.4.
Fixes bug 1226943
Change-Id: I23b804a4d99500b4acf81dd19645ab06dfcc9f1c
Previous _ was monkey patched into builtins whenever
certain modules were imported. This removes that and
simply imports it when it is needed.
Change-Id: I0af2c6d8a230e94440d655d13cab9107ac20d13c
The max header is exceeded in the following scenario
- Auth tokens built with a keystone v3 API catalog
- A catalog with approximately 8 or more endpoints defined
Specifically, when heat is enabled in tempest gating, the following
tests are failing:
- tempest.api.compute.test_auth_token.AuthTokenTestJSON.test_v3_token
- tempest.api.compute.test_auth_token.AuthTokenTestXML.test_v3_token
An equivalent fix will be required for all projects which use
eventlet.wsgi.
Fixes bug: #1190149
Change-Id: Ia0a2a48461593fdc0f5be57bcc9f364c3e3f8b66
Stephen Finucane