Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
some of them, about 10%.
Feel free to reject if we think it will cause too much
trouble with cherry-picks, else I'll slowly work my way
through the rest of the tree.
Trivialfix
Change-Id: I3d484d11e273cb8ee617f9445a069887e7b2b89f
Security folks considered tracebacks in API responses unwanted.
Some additional lower constraints had to be bumped for the
lower-constraints job to pass.
Change-Id: Ibaefbb9935020318ed670774b0205f3bcffef4ad
Closes-Bug: #1951429
Depends-On: https://review.opendev.org/c/openstack/oslo.service/+/818548
The function of class "ProcessLauncher" is to fork multiple processes.
the function of "WorkerService.start" is provider api service and set
process title according to the parameter desc. when multiple API
processes need to be launched, use "ProcessLauncher" (in module
oslo_service.service) manage. call "ProcessLauncher.launch_service"
with the num of processes and service instance to real launch. when the
"ProcessLauncher" launch the service, it calls the "start" method of
the service directly without passing any parameters.
Starting API processes, the service object passed in is WorkerService
(in module neutron.wsgi) object. In the current design, pass the "desc"
in call "WorkerService.start", therefore, the desc cannot be set.
This patch add the "desc" parameter to the initialization, when call
"start" method, if set "desc", it will be used first. if not, "desc"
passed in initialization are used. this design solves the problem and
keeps compatibility.
Closes-bug: #1910623
Change-Id: Ia37182e572ac4e9ae83cd03d7008aa42c7ea36c8
With python 3.x, six.text_type and six.string_type
are just str.
Also removed a six.integer_type since it was the only
one left in a file.
Another step in removing all of six usage from neutron.
Change-Id: I5208dc41bff1983ecd323286f427296b722da62a
If both are run under the same process, and api_workers >= 2, the server
process will instantiate two oslo_service.ProcessLauncher instances
This should be avoided [0], and indeed causes issues on subprocess and
signal handling: killed RPC workers not respawning, SIGHUP on master
process leading to unresponsive server, signal not properly sent to all
child processes, ...
To avoid this, use the wsgi ProcessLauncher instance if it exists
[0] https://docs.openstack.org/oslo.service/latest/user/usage.html#launchers
Change-Id: Ic821f8ca84add9c8137ef712031afb43e491591c
Closes-Bug: #1780139
We spawn a lot of neutron-servers, on all but the smallest systems.
It's often hard to tell which are busy/overloaded or spinning.
Add an option to set the process names to their role.
This has a small chance of breaking existing scripting, depending
how they're parsing ps output.
Sample output:
$ ps xw | grep neutron-server
1126 pts/2 S+ 0:00 grep --color=auto neutron-server
25355 ? Ss 0:26 /usr/bin/python /usr/local/bin/neutron-server \
--config-file /etc/neutron/neutron.conf \
--config-file /etc/neutron/plugins/ml2/ml2_conf.ini
25368 ? S 0:00 neutron-server: api worker
25369 ? S 0:00 neutron-server: api worker
25370 ? S 0:00 neutron-server: api worker
25371 ? S 0:00 neutron-server: api worker
25372 ? S 0:02 neutron-server: rpc worker
25373 ? S 0:02 neutron-server: rpc worker
25374 ? S 0:02 neutron-server: services worker
The "normal" looking ps output is the main parent.
Partial-Bug: #1816485
Depends-On: https://review.openstack.org/637119
Change-Id: I0e664a5f8e792d85b8f5483fb8c6f1cd59a677cd
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
The neutron.db.api.context_manager already references neutron-lib's
context manager; so consumers of it are already using neutron-lib. This
patch switches neutron's references to the context_manager over to
use neutron-lib's directly rather than that in neutron.db.api.
NeutronLibImpact
Change-Id: I97120faeec73690592ed21a5ec3c6202f61e1429
wsgi call to best_match() generates the following warning:
DeprecationWarning: The behavior of AcceptValidHeader.best_match
is currently being maintained for backward compatibility, but it
will be deprecated in the future, as it does not conform to the RFC.
Change to use acceptable_offers() as was done in nova code.
Simlar changes in the wsgi best language code as well, except
new call is to lookup().
Required bump to get WebOb 1.8.2.
Trivialfix
Change-Id: I45a222c098658eef6a49adcc168b0f5625ea49db
Fixed all pep8 E265 errors and changed tox.ini to no longer
ignore them. Also removed an N536 comment missed from a
previous change.
Change-Id: Ie6db8406c3b884c95b2a54a7598ea83476b8dba1
New releases of oslo.config support a 'mutable' parameter to Opts.
This is only respected when the new method mutate_config_files is
called instead of reload_config_files. Neutron delegates making this
call to oslo.service. This was provided in patchset
Icec3e664f3fe72614e373b2938e8dee53cf8bc5e
Further patches will be needed to make select config options be
marked as mutable. This change enables support for oslo provided
config options to be updated via SIGHUP such as log level.
Task: 6389
Story: 2001545
Change-Id: I9442965607f3248706464643c6d87a04edcae24e
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
neutron-lib 1.6 is out and among other things contains the rehomed
NeutronWorker class; now named BaseWorker in neutron-lib. This patch
updates references to use the lib version.
Since there's only 1 other consumer of these changes [1], this
patch depends on it so that it's free to merge after the single
consumer is updated.
[1] http://codesearch.openstack.org/?q=from%20neutron%20import%20worker
NeutronLibImpact
Change-Id: Id6bd414ae90119bf527bcaca6b88cc33ff79f685
Depends-On: Ifc0ecf6d5d3d4f8c406caad233d3648646724b12
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I8fb91bcb8ae4bfbfa68ddf7c38a4f1f5cf548b36
Currently, 'get_content_type' method in neutron/wsgi.py checks
specified format by 'in' statement.
Thus, string(e.g. 'application/j') that is partial matched to
'application/json' is returned.
However, we cannot find valid serializer from the format,
and request fails with unexpected error.
Change-Id: I75aac0308d0fd0321973c4bda3bc07ca0224e1c1
Closes-Bug: #1612485
Refactoring neutron wsgi config opts to be in neutron/conf so that all
the configuration options for wsgi reside in a centralized location.
This simplifies the process of looking up the wsgi config opts and
provides an easy way to import.
Change-Id: I6cf2617acd3932c60618821b6fb453c7a16e20cc
This reverts commit b1cdba1696
Original patch was reverted because it broke neutron plugin's
backward compatibility and needed more work.
This patch fixes that problems:
1) original behaviour of add_agent_status_check,
start_periodic_l3_agent_status_check and
start_periodic_dhcp_agent_status_check methods is deprecated but kept
for using in third part plugins for backward compatibility
2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
and add_periodic_dhcp_agent_status_check method are implemented
instead and are used for implementing plugins in neutron codebase
Closes-Bug: #1569404
Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6
I think this needs a little more thought. This broke OVN at least and
I don't think that's good. We need to figure out how to be compatible
with existing plugins, even out of tree ones.
This reverts commit 1cafff0871.
Change-Id: Ie087fb11213cc85911483c2d32c463fa9c973e54
Forking a process when multiple threads are running is an unsafe
operation and could cause a lot of problems because only current
thread will continue working in child thread. Any locked by other
thread resource will remain locked forever.
We faced with this problem during oslo.messaging development and
added workaround to hide this problem:
https://review.openstack.org/#/c/274255/
I tried to fix this problem in oslo.service:
https://review.openstack.org/#/c/270832/
but oslo folks said that this fix is ugly and it is wrong way to add
workarounds to common libraries because projects use them incorrectly.
I think that is fair.
So this patch fixes incorrect usage of oslo libraries. In this patch
I extended functionality of NeutronWorker and add there
`worker_process_count` parameter which determines how many processes
should be spawned for this worker. If `worker_process_count` = 0 - don't
create process and spawn thread in scope of current process for worker
Then I moved all background tasks to workers and return them by
`get_workers` method. start_plugin_workers collects plugin's workers
using `get_workers` method and starts in ProcessLauncher first workers
with `worker_process_count` > 0 and only after this starts threaded
workers by simple Launcher
Closes-bug: #1569404
Change-Id: I0544f1d47ae53d572adda872847a56fa0b202d2e
Neutron currently hardcodes the number of
greenlets used to process requests in a process to 1000.
As detailed in
http://lists.openstack.org/pipermail/openstack-dev/2015-December/082717.html
this can cause requests to wait within one process
for available database connection while other processes
remain available.
By adding a wsgi_default_pool_size option functionally
identical to that of Nova, we can lower the number of
greenlets per process to be more in line with a typical
max database connection pool size.
DocImpact: a previously unused configuration value
wsgi_default_pool_size is now used to affect
the number of greenlets used by the server. The
default number of greenlets also changes from 1000
to 100.
Change-Id: I94cd2f9262e0f330cf006b40bb3c0071086e5d71
Reactor code:
* Reuse oslo_utils.encodeutils.to_utf8() instead of existing
isinstance(text, six.text_type) test
* Replace jsonutils.dumps(obj).encode('utf-8') with
jsonutils.dump_as_bytes(obj).
* Other minor bytes/Unicode changes
Change-Id: I03b8eff0fd70ab65ac66d6f3221e8ced0a56db17
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
The communication between the ns metadata proxy and the metadata agent
is pure HTTP, and should not switch to HTTPS when neutron is using SSL.
We're therefore telling wsgi.Server to forcefully disable SSL in that
case.
Change-Id: I2cb9fa231193bcd5c721c4d5cf0eb9c16e842349
Closes-Bug: #1514424
Went through all of the docstrings in Neutron and did
some cleanup. I'm sure there are bunch more that I have missed.
Change-Id: Ib29d2de1c580880c89ed4fd069e1515d0977a3e7
oslo_service 0.10.0 release added register_opts function to wsgi
module that is why it is no longer needed to access its private
_options module.
Change-Id: I4803a3540a6bc9c8f08db49e209668d0736abec7
Depends-On: Id0f9f93758c68aa954d37e3566e35deae2961268
Middleware class in neutron/wsgi.py can be replaced with
ConfigurableMiddleware from oslo_middleware.
Remove the Debug helper class as it doesn't seem to be used
anywhere. Instead, one can use oslo_middleware.debug.Debug.
Added a note about possible implications for out-of-tree plugins
to neutron_api.rst.
Change-Id: If7360608f94625b7d0972267b763f3e7d7624fee
sslutils and basic WSGI functionality have been moved to
oslo.service and now Neutron can reuse them.
Marked ssl options that were renamed in oslo.service as
deprecated.
Added a note about possible implications for out-of-tree plugins
to neutron_api.rst
Bumped oslo.service version to 0.9.0.
Related-Bug: #1482633
Depends-On: I0424a6c261fae447dbc25b3abf00258c860a88f5
Change-Id: Ibfdf07e665fcfcd093a0e31274e1a6116706aec2
With Eventlet 0.17.2, we can directly pass the logger instance. This
allows us to deprecate/remove the WritableLogger class from oslo.log.
Closes-Bug: #1440773
Change-Id: I3985b30b499a1e676d39c2a6cd3a31ac13c0ab47
There are several cases where plugin initialization should be
handled after neutron-server forks API/RPC workers. For example,
starting a client connection to an SDN controller before forking
copies the fd of the socket to the child process, but then you have
multiple processes trying to read/write the same socket connection.
It is also useful for a plugin to be able to do something in only
one process, regardless of how many workers are forked. One example
would be handling syncing from an external system to the neutron
database.
This patch does 3 things:
1) Treats rpc_workers=0 as = 1. This simplifies the code for
handling notification that forking has completed. In the
existing code, calling the notification in the Worker object's
start() method would happen twice in the case where both api
and rpc workers were 0, despite there being only one process.
An earlier patch already changed the default api_workers to be
the number of processors.
2) Adds notification of forking via the callbacks mechanism.
Plugins can subscribe to resources.PROCESS, event.AFTER_CREATE
and do any post-fork initialization that needs to be done for
every spawned process.
3) Adds core/service plugin calls to get_workers() which defaults
to returning (). Plugins that need additional processes to spawn
should just return an iterable of NeutronWorkers that will be
spawned in their own process.
DocImpact
Closes-Bug: #1463129
Change-Id: Ib99954678c2b4f32f486b537979d446aafbea07b
WebOb disallows in py3K to set webob.Response.body to a unicode object,
we should encode unicode bodies in such case.
Change-Id: Ie0dc57fbe3ed9b19dac2e958de14387bc4c1a260
Blueprint: neutron-python3
The constants of log levels were added in the 1.8 version of
the oslo.log library. So we can replace all usage of system
logging module with log module from oslo.log.
Change-Id: I2992df0bec6337aefa8a75d4853b132bd134fa42
Closes-Bug: 1481370
This patch ensures log_opt_values is invoked only once, thus
avoiding annoying multiple dumps of options values.
Note that is rpc_workers is greater than 0 a second dump will
be logged. This patch does not address this as REST and RPC
servers are going to be separated as a part of the feature/pecan
work.
Closes-Bug: #1477975
Change-Id: Ia5dcb609241de6ad30d9831c5fb98a9e2be6ad7f
oslo.service has graduated, so neutron should consume it.
Closes-Bug: #1466851
Depends-On: Ie0fd63f969f954029c3c3cf31337fbe38f59331a
Depends-On: I2093b37d411df9a26958fa50ff523c258bbe06ec
Depends-On: I4823d344878fc97e66ddd8fdae25c13a34dede40
Change-Id: I0155b3d8b72f6d031bf6f855488f80acebfc25d4
In Python 3, 'unicode' does not exist; 'six.text_type' should be used instead.
Change-Id: I71011b4beee9817a61278eb473804cfb798de74a
Blueprint: neutron-python3
All launchers implemented in common.service require each service to
implement reset method because it is called in case a process
receives a SIGHUP.
This change adds the reset method to neutron.service.RpcWorker and
neutron.wsgi.WorkerService which are used to wrap rpc and api
workers correspondingly.
Now neutron-server running in multiprocess mode (api_workers > 0 and
rpc_workers > 0) and metadata agent don't die on receiving SIGHUP and support
reloading policy_path and logging options in config.
Note that reset is called only in case a service is running in daemon mode.
Other changes made in the scope of this patch that need to be mentioned:
* Don't empty self._servers list in RpcWorker's stop method
When a service is restarted all services are gracefully shutdowned,
resetted and started again (see openstack.common.service code).
As graceful shutdown implies calling service.stop() and then
service.wait() we don't want to clean self._servers list because
it would be impossible to wait for them to stop processing
requests and cleaning up their resources.
Otherwise, this would lead to problems with rpc after starting
the rpc server again.
* Create a duplicate socket each time WorkerService starts
When api worker is stopped it kills the eventlet wsgi server
which internally closes the wsgi server socket object. This server
socket object becomes not usable which leads to "Bad file
descriptor" errors on service restart.
Added functional and unit tests.
DocImpact
Partial-Bug: #1276694
Change-Id: I75b00946b7cae891c6eb192e853118e7d49e4a24
Brian Haley