Remove B303 (md5, sha1 for python<3.9) and
remove B311 (Standard pseudo-random generators are
not suitable for security/cryptographic purpose) from
the skip list of bandit execution.
Change-Id: I6e9e61e7f94dc9ca339942529af8997adef45e38
Currently at least 1 rpc worker is launched even when a user requests
zero workers by setting rpc_workers=0. The setting of rpc_workers=0 is
used when ml2-ovn plugin is used without any additional agent, and in
this deployment pattern the single rpc worker is not at all used.
This change ensures no rpc worker is launched when rpc_workers options
is explicitly set to 0. This may be classified as a breaking change,
but is consistent with the earlier change[1] for rpc_workers=0.
[1] 3e1e2d63b3
Closes-Bug: #2052484
Change-Id: I878e50c3192ecd3b145ded0ab8394845a089696e
Currently both RpcReportsWorker and RpcWorker are launched with
the same process title("rpc worker"), and we can't distinguish one from
the other.
This changes the process title of RpcReportsWorker, so that we can
find the worker used by RpcReportsWorker, not by RpcWorker.
Change-Id: I610a00667762bbdd45bc72c5a865694b92cfd45a
Currently logger is not properly initialized according to config
options or process arguments when the neutron-rpc-server script or
the api wsgi script is used. Because of this neutron does not generate
the log files as expected when we deploy api using WSGI and use
neutron-rpc-server, instead of the monolithic neutron-server.
This ensures the logger is initialized even when these two scripts
are used.
Closes-Bug: #2030904
Related-Bug: #1693691
Change-Id: I13bcd36c237555e0ca60d5ffc214b8f3bf6e478f
After updating pylint, it started emitting additional "R"
warnings in some cases, fix some of them.
use-a-generator,
unnecessary-lambda-assignment,
consider-using-max-builtin,
consider-using-generator,
consider-using-in,
use-list-literal,
consider-using-from-import
Trivialfix
Change-Id: Ife6565cefcc30b4e8a0df9121c9454cf744225df
If both are run under the same process, and api_workers >= 2, the server
process will instantiate two oslo_service.ProcessLauncher instances
This should be avoided [0], and indeed causes issues on subprocess and
signal handling: killed RPC workers not respawning, SIGHUP on master
process leading to unresponsive server, signal not properly sent to all
child processes, ...
To avoid this, use the wsgi ProcessLauncher instance if it exists
[0] https://docs.openstack.org/oslo.service/latest/user/usage.html#launchers
Change-Id: Ic821f8ca84add9c8137ef712031afb43e491591c
Closes-Bug: #1780139
The help for the rpc_workers config option is:
Number of RPC worker processes for service. If not specified, the
default is equal to half the number of API workers.
However, this does not accurately describe the current behaviour,
which is to default to half the _default_ number of API workers. This
can make a big difference; for example on a 256-CPU machine with 256GB
of RAM which has api_workers configured to 8 but rpc_workers not
configured to anything, this will result in 64 RPC workers, which is 8
for every API worker!
Therefore tweak the default to rely on the actual value of
api_workers, which may be different than the default value.
Change-Id: I26115932ef4775f157297be1637ee26a4fca4666
Related-Bug: #1838688
Closes-Bug: #1838689
We spawn a lot of neutron-servers, on all but the smallest systems.
It's often hard to tell which are busy/overloaded or spinning.
Add an option to set the process names to their role.
This has a small chance of breaking existing scripting, depending
how they're parsing ps output.
Sample output:
$ ps xw | grep neutron-server
1126 pts/2 S+ 0:00 grep --color=auto neutron-server
25355 ? Ss 0:26 /usr/bin/python /usr/local/bin/neutron-server \
--config-file /etc/neutron/neutron.conf \
--config-file /etc/neutron/plugins/ml2/ml2_conf.ini
25368 ? S 0:00 neutron-server: api worker
25369 ? S 0:00 neutron-server: api worker
25370 ? S 0:00 neutron-server: api worker
25371 ? S 0:00 neutron-server: api worker
25372 ? S 0:02 neutron-server: rpc worker
25373 ? S 0:02 neutron-server: rpc worker
25374 ? S 0:02 neutron-server: services worker
The "normal" looking ps output is the main parent.
Partial-Bug: #1816485
Depends-On: https://review.openstack.org/637119
Change-Id: I0e664a5f8e792d85b8f5483fb8c6f1cd59a677cd
The configuration variable "rpc_response_max_timeout" is not defined
in the OVS agent configuration. When the agent is stopped (SIGTERM),
the exception is raised. This error can be seen in the fullstack tests.
Change-Id: Ieedb6e87a4e98efef0f895566f7d4d88c3cd9336
Closes-Bug: #1815797
- Limit number of api workers to roughly using half of system
RAM. Spawning a bunch, just to have the OOM killer nuke them
regularly is not useful.
- Bump the rpc_workers default to half of the api_workers.
A default of 1 falls behind on any reasonably sized node.
Change-Id: I8b84a359f83133014b3d4414aafc10e6b7c6a876
Closes-bug: #1815629
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
NeutronLibImpact
Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
The context_manager is already in neutron-lib and being used by our
active consumers either directly or via the current shim we have
in neutron.db.api.context_manager that just references that in lib.
This patch removes the context_manager from neutron and updates the UTs
that mock it.
NeutronLibImpact
Change-Id: Ia5191a71b377e2862592a09da8dc3a5d415de9aa
neutron-api needs to load config files when deployed under
WSGI process. This patch will do this. neutron-rpc-server
also need to be aware of plugin extension.
uWSGI seems not happy with eventlet.GreenPool. So this patch
will switch to futurist.ThreadPoolExcutor for ML2 OVO push
notification.
Partially-implements: blueprint run-in-wsgi-server
Change-Id: Ia2659c9c8f1deeceefc1c75f1149a1e47c91e7a9
Closes-Bug: #1708389
New releases of oslo.config support a 'mutable' parameter to Opts.
This is only respected when the new method mutate_config_files is
called instead of reload_config_files. Neutron delegates making this
call to oslo.service. This was provided in patchset
Icec3e664f3fe72614e373b2938e8dee53cf8bc5e
Further patches will be needed to make select config options be
marked as mutable. This change enables support for oslo provided
config options to be updated via SIGHUP such as log level.
Task: 6389
Story: 2001545
Change-Id: I9442965607f3248706464643c6d87a04edcae24e
This patch switches callbacks over to the payload object style events
[1] for BEFORE_SPAWN and AFTER_SPAWN based notifications. As no event
data is passed for SPAWN notifications, an actual payload object is
not needed and thus this change uses publish() rather than notify()
and updates and callback receivers to accept the payload parameter.
NeutronLibImpact
[1] https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html#event-payloads
Change-Id: Ifad4781d82aa07f213f4e075b0c9455aa95e94bb
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
neutron-lib 1.6 is out and among other things contains the rehomed
NeutronWorker class; now named BaseWorker in neutron-lib. This patch
updates references to use the lib version.
Since there's only 1 other consumer of these changes [1], this
patch depends on it so that it's free to merge after the single
consumer is updated.
[1] http://codesearch.openstack.org/?q=from%20neutron%20import%20worker
NeutronLibImpact
Change-Id: Id6bd414ae90119bf527bcaca6b88cc33ff79f685
Depends-On: Ifc0ecf6d5d3d4f8c406caad233d3648646724b12
Add callback (PRCESS, AFTER_SPAWN) to do housekeeping task.
ML2 driver(in fact, networking-odl) wants to run house keeping task
which needs to run only in api server.
The following is very netwokring-odl specific as FYI:
The task is to run a timer which is extended when rest requests are
handled. The timer is fired to kick journal thread only when rest
request is idle in order to check if there is journal entries that was
created by other neutron servers that crash before processing them.
Such timer isn't needed for each workers. Only single timer among api
worker and rpc workers is enough.
cf https://review.openstack.org/#/c/461620/
The current approach is to run timer unconditionally for all processes
and stop it if it's worker process. With this patch, the timer can be
simply run for main process(api worker).
Change-Id: I9c07bc528c3a2fade0c835797889fc169f9bd1a6
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
The current PROCESS AFTER_INIT event (or AFTER_CREATE) are being called
multiple times, once for each process.
In case we want to perform an action only once after init, we would need a
callback that is called after init is done, but before spawning.
Usage examples can be:
- A plugin/driver needs to configure something at the backend after init is done
- A plugin/driver needs to create an object in one of the neutron tables
(add a network for example)
Change-Id: I249ba32dede16e61b1c757a8010dac610484da4f
This reverts commit b1cdba1696
Original patch was reverted because it broke neutron plugin's
backward compatibility and needed more work.
This patch fixes that problems:
1) original behaviour of add_agent_status_check,
start_periodic_l3_agent_status_check and
start_periodic_dhcp_agent_status_check methods is deprecated but kept
for using in third part plugins for backward compatibility
2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
and add_periodic_dhcp_agent_status_check method are implemented
instead and are used for implementing plugins in neutron codebase
Closes-Bug: #1569404
Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6
The option was deprecated a long time ago, and will be removed in one of
the next library releases, which will render neutron broken if we keep
using the option.
More details:
http://lists.openstack.org/pipermail/openstack-dev/2016-May/095166.html
Closes-Bug: #1586066
Change-Id: I884b4cc3ed04e4b5489e265c146666e04eb1bc27
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
1) It checks that person who wants to trace is trusted and knows
secret HMAC key.
2) It starts tracing in case of proper trace headers
and adds first wsgi trace point, with info about HTTP request
* Add initialization of osprofiler at start of service
Currently that includes oslo.messaging notifer instance creation
to send Ceilometer backend notifications.
Neutron client change: Ic11796889075b2a0e589b70398fc4d4ed6f3ef7c
Co-authored-by: Ryan Moats <rmoats@us.ibm.com>
Depends-On: I5102eb46a7a377eca31375a0d64951ba1fdd035d
Closes-Bug: #1335640
DocImpact Add devref and operator documentation on how to use this
APIImpact
Change-Id: I7fa2ad57dc5763ce72cba6945ebcadef2188e8bd
Refactoring neutron configuration options for service to be in
neutron/conf. This would allow centralization of all configuration
options and provide an easy way to import.
Co-Authored-By: Dariusz Smigiel <smigiel.dariusz@gmail.com>
Partial-Bug: #1563069
Change-Id: Iff140c11765199536a8fd8066ad3063ed92355ae
I think this needs a little more thought. This broke OVN at least and
I don't think that's good. We need to figure out how to be compatible
with existing plugins, even out of tree ones.
This reverts commit 1cafff0871.
Change-Id: Ie087fb11213cc85911483c2d32c463fa9c973e54
Forking a process when multiple threads are running is an unsafe
operation and could cause a lot of problems because only current
thread will continue working in child thread. Any locked by other
thread resource will remain locked forever.
We faced with this problem during oslo.messaging development and
added workaround to hide this problem:
https://review.openstack.org/#/c/274255/
I tried to fix this problem in oslo.service:
https://review.openstack.org/#/c/270832/
but oslo folks said that this fix is ugly and it is wrong way to add
workarounds to common libraries because projects use them incorrectly.
I think that is fair.
So this patch fixes incorrect usage of oslo libraries. In this patch
I extended functionality of NeutronWorker and add there
`worker_process_count` parameter which determines how many processes
should be spawned for this worker. If `worker_process_count` = 0 - don't
create process and spawn thread in scope of current process for worker
Then I moved all background tasks to workers and return them by
`get_workers` method. start_plugin_workers collects plugin's workers
using `get_workers` method and starts in ProcessLauncher first workers
with `worker_process_count` > 0 and only after this starts threaded
workers by simple Launcher
Closes-bug: #1569404
Change-Id: I0544f1d47ae53d572adda872847a56fa0b202d2e
When neutron starts an rpc worker, it checks if the plugin has the method
"start_rpc_listeners". Since most plugins inherit from a base class, and
that base class implements the start_rpc_listeners method and raises
NotImplementedError, the rpc worker will attempt to call that method.
It should just catch the NotImplementedError and continue on.
Change-Id: Ie1830b6140acffffd0f283a0d8eefa52067f7650
Closes-Bug: 1551542
Use the same wsgi server as the homegrown method dispatch
so we get support for api_workers and plugin workers when
using pecan.
Partially-Implements: wsgi-pecan-switch
Change-Id: I9fb942651d9b52b48e8cd80b0a6ab77ec37ea1e8
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
By default spawn one additional rpc worker to process
state report queue.
State report queue will also be processed by regular
rpc workers, but in case these workers are busy with
processing heavy requests, state reports queue will
automatically be consumed by dedicated rpc workers.
This change applies to ML2 plugin only.
Other plugins should implement start_rpc_state_reports_listener
to enable additional rpc workers.
Change-Id: I5f8df6a478f7c82382049274b34b07109eeafbdb
Closes-Bug: #1505217
Patch 5be613490d erroneously removed this
line that is still needed which broke networking-ovn, at least.
Change-Id: I31981e905f892df474c57b45d3e70ebfbeeae1db
Signed-off-by: Russell Bryant <rbryant@redhat.com>
This patch adds all RPC workers to consumers of service
plugins queues such as metering and l3-plugin.
This is important for DVR-enabled deployments with hundreds
of agents.
Change-Id: I6fea7f409c91b25d2c35b038d6100fdfa85d1905
Closes-Bug: #1498844
There are several cases where plugin initialization should be
handled after neutron-server forks API/RPC workers. For example,
starting a client connection to an SDN controller before forking
copies the fd of the socket to the child process, but then you have
multiple processes trying to read/write the same socket connection.
It is also useful for a plugin to be able to do something in only
one process, regardless of how many workers are forked. One example
would be handling syncing from an external system to the neutron
database.
This patch does 3 things:
1) Treats rpc_workers=0 as = 1. This simplifies the code for
handling notification that forking has completed. In the
existing code, calling the notification in the Worker object's
start() method would happen twice in the case where both api
and rpc workers were 0, despite there being only one process.
An earlier patch already changed the default api_workers to be
the number of processors.
2) Adds notification of forking via the callbacks mechanism.
Plugins can subscribe to resources.PROCESS, event.AFTER_CREATE
and do any post-fork initialization that needs to be done for
every spawned process.
3) Adds core/service plugin calls to get_workers() which defaults
to returning (). Plugins that need additional processes to spawn
should just return an iterable of NeutronWorkers that will be
spawned in their own process.
DocImpact
Closes-Bug: #1463129
Change-Id: Ib99954678c2b4f32f486b537979d446aafbea07b
It looks like recent changes to oslo.messaging master are conflicting
with changes in neutron master with the way RPC services are started
when the rpc_workers value == 0.
Change-Id: Iea2197ad0ea9ceb9a2a850a9e03e53b4b39ca288
This patch ensures log_opt_values is invoked only once, thus
avoiding annoying multiple dumps of options values.
Note that is rpc_workers is greater than 0 a second dump will
be logged. This patch does not address this as REST and RPC
servers are going to be separated as a part of the feature/pecan
work.
Closes-Bug: #1477975
Change-Id: Ia5dcb609241de6ad30d9831c5fb98a9e2be6ad7f
oslo.service has graduated, so neutron should consume it.
Closes-Bug: #1466851
Depends-On: Ie0fd63f969f954029c3c3cf31337fbe38f59331a
Depends-On: I2093b37d411df9a26958fa50ff523c258bbe06ec
Depends-On: I4823d344878fc97e66ddd8fdae25c13a34dede40
Change-Id: I0155b3d8b72f6d031bf6f855488f80acebfc25d4
This reverts commit 12a564cf03.
We should re-enable this feature on a controlled basis so that we can
flush out any outstanding issue we may have.
Related-bug: #1432189
Change-Id: I2cfd93fdb032b461022b729347390ff8636ccdeb
elajkat