Commit Graph

50 Commits

Author SHA1 Message Date
Zuul 46ba0fa339 Merge "Register GMR with config" 2023-12-08 18:32:17 +00:00
Takashi Kajinami 6a5859fc07 Fix python shebang
The current shebang requires /usr/bin/python which is not available in
Ubuntu Jammy by default.

This also fixes some unnecessary/missing shebangs.

Change-Id: Ib25a0a7f39f68f43622609391710dd3b1abc2d00
2023-10-17 16:02:25 +00:00
Arnaud Morin 56821b6aea Register GMR with config
As an operator, we want to set oslo_reports/file_event_handler because
when running behind a wsgi server, GMR cannot register the Signal to be
trigger.
The parameter file_event_handler has been designed for this specific use
case but it was not used correctly by neutron.

Closes-Bug: #2021814

Change-Id: Id13de1a3f9ea2eaaa7521eedf905aa0dd993ff89
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
2023-09-26 14:56:06 +00:00
Takashi Kajinami afcce6d749 Initialize logger for rpc-server and wsgi script
Currently logger is not properly initialized according to config
options or process arguments when the neutron-rpc-server script or
the api wsgi script is used. Because of this neutron does not generate
the log files as expected when we deploy api using WSGI and use
neutron-rpc-server, instead of the monolithic neutron-server.

This ensures the logger is initialized even when these two scripts
are used.

Closes-Bug: #2030904
Related-Bug: #1693691
Change-Id: I13bcd36c237555e0ca60d5ffc214b8f3bf6e478f
2023-08-10 02:03:53 +00:00
Brian Haley 07fd743dc9 Remove extra call to config_logging() for neutron-server
Logging is setup early in neutron/service.py already.

Change-Id: Ib1b42235032a179a77932b6e3049f50c41073bfd
Closes-bug: #1693691
2023-02-02 02:25:00 +00:00
labedz dcbc4beb9e Use monkey_patch for neutron API server.
Neutron API server was not using eventlet monkey patch
thus eventlet threads couldn't be properly yielded.

This patch set API neutron server like other monkey patched
neutron services: neutron-server and neutron-rpc-server.

NOTE: this change needs apache service restart. Apache
mod_wsgi auto reload can lead to SSL RecursionError.

Co-Authored-By: Szymon Wroblewski <szymon.wroblewski@ovhcloud.com>
Closes-Bug: 1970216
Change-Id: Ib62c049cc521a548ab7e7e9584b19bdaa67b1c9d
2022-04-28 13:34:36 +00:00
Jakub Libosvar 4d3a274765 Don't register config options on imports
Importing some modules lead to registering config options that may
collide with config options from a project that calls the import. This
patch wraps the side effect that registers config options into a
function that needs to be called in case the caller wants to register
the options.

This solution is also not perfect as it guards the common options to be
registered only once even if the function is called multiple times. This
is to solve problems in unittests, ideally we should always call the
function just once even in our testing suites.

Resolves-Bug: #1968606
Change-Id: Ic1532eb8de887ff1b1085206df11f53e22f7f524
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
2022-04-13 05:49:15 +00:00
Bernard Cafarelli 13aa00026f
Re-use existing ProcessLauncher from wsgi in RPC workers
If both are run under the same process, and api_workers >= 2, the server
process will instantiate two oslo_service.ProcessLauncher instances

This should be avoided [0], and indeed causes issues on subprocess and
signal handling: killed RPC workers not respawning, SIGHUP on master
process leading to unresponsive server, signal not properly sent to all
child processes, ...

To avoid this, use the wsgi ProcessLauncher instance if it exists

[0] https://docs.openstack.org/oslo.service/latest/user/usage.html#launchers

Change-Id: Ic821f8ca84add9c8137ef712031afb43e491591c
Closes-Bug: #1780139
2020-02-07 14:51:06 +01:00
Nguyen Phuong An 2f98f56cb5 Allow neutron-api load config from WSGI process
neutron-api needs to load config files when deployed under
WSGI process. This patch will do this. neutron-rpc-server
also need to be aware of plugin extension.

uWSGI seems not happy with eventlet.GreenPool. So this patch
will switch to futurist.ThreadPoolExcutor for ML2 OVO push
notification.

Partially-implements: blueprint run-in-wsgi-server

Change-Id: Ia2659c9c8f1deeceefc1c75f1149a1e47c91e7a9
Closes-Bug: #1708389
2018-07-25 15:22:14 +07:00
Inessa Vasilevskaya 7322bd6efb Make code follow log translation guideline
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.

Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
2017-08-14 02:01:48 +00:00
Kevin Benton 57a37c4a87 Switch to start_all_workers in RPC server
This does the same as the logic present but it emits
the registry callback event for resources.PROCESS AFTER_SPAWN
that some plugins may be expecting.

Change-Id: I6f9aeca753a5d3c0052f553a2ac46786ca113e1e
Related-Bug: #1687896
2017-06-13 02:20:35 -07:00
Sean McCully 0ef8e2c168 neutron-rpc-server fails with no plugins loaded
Neutron RPC Server fails with exception
"object has no attribute 'rpc_workers_supported'".

Change-Id: I0997789bb61635ed36e4b95f03fc7d396511e19f
Closes-Bug: #1687896
2017-06-06 20:31:42 -04:00
Victor Morales 429703a9a8 Expose neutron api application as a wsgi script
This can then be loaded into a wsgi compatible web server like
apache2/httpd or nginx.

DocImpact A new neutron-api script is installed, and can be used to
start either a standalone wsgi server (calling the script directly),
or as a wsgi script loaded by a web server (e.g. by httpd/mod_wsgi).

Change-Id: Ie9bb8b33f6e1335219e574d832ac264b043d90e6
Partial-Bug: #1666779
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Co-Authored-By: Ihar Hrachyshka <ihrachys@redhat.com>
2017-04-17 02:07:27 +00:00
tonytan4ever ebc7e1fb2f Pecan: Get loaded by paste deploy
This sets up the factory methods needed to have paste deploy create the
pecan app and return it.  It also changes the legacy factory methods to
conditionally use the pecan factory methods if the web_framework config
option is set to 'pecan'.  This way, all deployments of neutron will not
need to change their api-paste.ini files to get pecan toggled on.  It
should just happen without notice once pecan becomes the default.

Also, by moving this to be loaded by paste deploy, there is a good chunk of
code that has been removed because it is no longer necessary.

Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Change-Id: I8b1bbea8d90fdc62715cd8b6738ad955df53d7cd
2017-02-17 14:54:27 -06:00
Dmitriy Ukhlov 483c5982c0 Revert "Revert "Remove threading before process forking""
This reverts commit b1cdba1696

Original patch was reverted because it broke neutron plugin's
backward compatibility and needed more work.

This patch fixes that problems:
1) original behaviour of add_agent_status_check,
   start_periodic_l3_agent_status_check and
   start_periodic_dhcp_agent_status_check methods is deprecated but kept
   for using in third part plugins for backward compatibility
2) new add_agent_status_check_worker, add_periodic_l3_agent_status_check
   and add_periodic_dhcp_agent_status_check method are implemented
   instead and are used for implementing plugins in neutron codebase

Closes-Bug: #1569404

Change-Id: I3a32a95489831f0d862930384309eefdc881d8f6
2016-06-02 21:26:08 +00:00
Carl Baldwin b1cdba1696 Revert "Remove threading before process forking"
I think this needs a little more thought.  This broke OVN at least and
I don't think that's good.  We need to figure out how to be compatible 
with existing plugins, even out of tree ones.

This reverts commit 1cafff0871.

Change-Id: Ie087fb11213cc85911483c2d32c463fa9c973e54
2016-05-05 21:07:21 +00:00
dukhlov 1cafff0871 Remove threading before process forking
Forking a process when multiple threads are running is an unsafe
operation and could cause a lot of problems because only current
thread will continue working in child thread. Any locked by other
thread resource will remain locked forever.

We faced with this problem during oslo.messaging development and
added workaround to hide this problem:
https://review.openstack.org/#/c/274255/

I tried to fix this problem in oslo.service:
https://review.openstack.org/#/c/270832/

but oslo folks said that this fix is ugly and it is wrong way to add
workarounds to common libraries because projects use them incorrectly.
I think that is fair.

So this patch fixes incorrect usage of oslo libraries. In this patch
I extended functionality of NeutronWorker and add there
`worker_process_count` parameter which determines how many processes
should be spawned for this worker. If `worker_process_count` = 0 - don't
create process and spawn thread in scope of current process for worker

Then I moved all background tasks to workers and return them by
`get_workers` method. start_plugin_workers collects plugin's workers
using `get_workers` method and starts in ProcessLauncher first workers
with `worker_process_count` > 0 and only after this starts threaded
workers by simple Launcher

Closes-bug: #1569404

Change-Id: I0544f1d47ae53d572adda872847a56fa0b202d2e
2016-05-03 21:54:23 +03:00
Michael Krotscheck aae181817e Moved CORS middleware configuration into oslo-config-generator
The default values needed for neutron's implementation of cors
middleware have been moved from paste.ini into the configuration
hooks provided by oslo.config. Furthermore, these values have been
added to neutron's default configuration parsing. This ensures
that if a value remains unset in neutron.conf, it will be set to use
sane defaults, and that an operator modifying the configuration
file will be presented with a default set of necessary sane headers.

Change-Id: I327d1d7ce50dd4a20d781e6251a5af81c57e6ca1
Closes-Bug: 1551836
2016-03-01 12:21:33 -08:00
Jenkins c2139fdecb Merge "Support api_workers option when using pecan" 2016-02-04 18:49:44 +00:00
Jenkins 7fca441ce1 Merge "RPC worker support for pecan server" 2016-02-04 18:49:13 +00:00
Jenkins b80d77ddfe Merge "Don't decide web_framework before config parse" 2016-02-04 04:09:38 +00:00
Kevin Benton f37079592b Support api_workers option when using pecan
Use the same wsgi server as the homegrown method dispatch
so we get support for api_workers and plugin workers when
using pecan.

Partially-Implements: wsgi-pecan-switch
Change-Id: I9fb942651d9b52b48e8cd80b0a6ab77ec37ea1e8
2016-01-26 13:13:46 -08:00
Kevin Benton 3d84277897 RPC worker support for pecan server
This patch adds support for launching the RPC worker processes
when using the simple wsgi server for pecan.

Change-Id: I7274083c32fd57133f8b33241c5fa95a8bb9bbb3
Partially-Implements: wsgi-pecan-switch
2016-01-22 17:44:38 -08:00
Kevin Benton d21d9798fe Don't decide web_framework before config parse
This ensures that the config files passed on the CLI
are parsed before the choice is made based on config options
to launch pecan/homegrown wsgi handler.

Closes-Bug: #1537292
Change-Id: I577befea74d84d611cd198c92c54ae3651e2d921
2016-01-22 17:44:35 -08:00
Bhagyashri Shewale 88e899f7a0 Fix module's import order
Made corrections in import order for built-in, third party and
project specific modules as per OpenStack import standards [1].

[1] http://docs.openstack.org/developer/hacking/#import-order-template

Change-Id: I899deefd6ee4732d6c0afd17a5afbe42b0fa37ba
2016-01-22 06:38:42 -08:00
Akihiro Motoki 2d8632e412 Use _ from neutron._i18n
Partial-Bug: #1520094
Change-Id: I874a4aa1d71d1f7034a1ff0b7450b419ef5c6864
2015-12-06 19:39:04 +09:00
Doug Wiegley dd726ed494 Move i18n to _i18n, as per oslo_i18n guidelines
- This does NOT break other projects that rely on neutron.i18n,
  as this change includes a debtcollector shim to maintain those
  older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
  http://docs.openstack.org/developer/oslo.i18n/usage.html

Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
2015-12-01 19:29:10 -07:00
Jenkins 50dac76b06 Merge "Introduce a separate RPC server" into feature/pecan 2015-09-18 16:58:54 +00:00
Salvatore Orlando fb69960f1e Introduce a separate RPC server
As the Pecan server only server REST requests over HTTP, this
patch introduces a new server implementing the RPC over AMQP
endpoints for agent/server communication.

However, the REST server does not yet have the ability to send
notifications to the RPC server or directly to the agents.
This patch simply adapts the ML2 plugin to run the RPC notifiers
only when initialized in the pecan server, so that notification
to agents can still be sent.

This patch therefore is tantamount to a poor man's
implementation of REST/RPC separation which will be iteratively
improved.

Change-Id: Ie471869d9b2793acdc412f13507038433f6a72c6
2015-09-17 05:18:32 -07:00
Kevin Benton 40151be9b3 Revert "Revert "Pecan WSGI: prevent plugins from opening AMQP connections""
This reverts commit 4f1fb05959.
Merge of master went in so this can be re-applied and have the
conflict resovled.

Conflicts:
	neutron/server/__init__.py

Change-Id: I917e89b2191b3d4646e66a7a313783a56cf0ace5
2015-09-16 03:33:47 -07:00
Terry Wilson 9f6bd17703 Add support for PluginWorker and Process creation notification
There are several cases where plugin initialization should be
handled after neutron-server forks API/RPC workers. For example,
starting a client connection to an SDN controller before forking
copies the fd of the socket to the child process, but then you have
multiple processes trying to read/write the same socket connection.

It is also useful for a plugin to be able to do something in only
one process, regardless of how many workers are forked. One example
would be handling syncing from an external system to the neutron
database.

This patch does 3 things:
1) Treats rpc_workers=0 as = 1. This simplifies the code for
   handling notification that forking has completed. In the
   existing code, calling the notification in the Worker object's
   start() method would happen twice in the case where both api
   and rpc workers were 0, despite there being only one process.
   An earlier patch already changed the default api_workers to be
   the number of processors.
2) Adds notification of forking via the callbacks mechanism.
   Plugins can subscribe to resources.PROCESS, event.AFTER_CREATE
   and do any post-fork initialization that needs to be done for
   every spawned process.
3) Adds core/service plugin calls to get_workers() which defaults
   to returning (). Plugins that need additional processes to spawn
   should just return an iterable of NeutronWorkers that will be
   spawned in their own process.

DocImpact

Closes-Bug: #1463129
Change-Id: Ib99954678c2b4f32f486b537979d446aafbea07b
2015-09-03 06:40:50 +00:00
Kevin Benton be9bff66be Rename 'newapi' to 'pecan_wsgi'
'newapi' was too generic and not really descriptive of the conversion
to pecan. Since there isn't any ambiguity, we will just keep all of
the pecan components in a directory called 'pecan_wsgi'.

Change-Id: I61f1cfcad4ff00582e0a481400642dd656d5245e
2015-09-02 15:53:34 +00:00
Doug Wiegley 2c5f44e1b3 Merge remote-tracking branch 'origin/master' into merge-branch
Change-Id: I9c29ad9564671ae5a2db35835bc4a30e75482cb2
2015-08-30 21:23:59 -06:00
Salvatore Orlando 2ba2456f9d Pecan WSGI: prevent plugins from opening AMQP connections
The Pecan WSGI server is supposed to server REST requests only.
As several plugins initialize their RPC interfaces upon initialization,
this results in establishing undesired connections to the AMQP server.

This patch does not alter plugin behavior or initialization process,
but ensures that the setup_rpc operation performed by a plugin has no
effect when executed in the Pecan WSGI server.

This patch also performs some refactoring for server launchers.

Change-Id: I56384f5f964ea90d72babf911aa5639989e9c3d8
2015-08-13 15:10:48 -07:00
Cedric Brandily afd08e066c Remove unneeded shebangs
This change removes unneeded shebangs in python modules.

Change-Id: Id0a3664eddf1ee26830998f4c73e89d0de5249d6
2015-08-13 12:39:07 +02:00
Qiaowei Ren 78ff49ecf9 Remove unnecessary executable permission
Bunch of neutron source code files are marked as executable which is not
appropriate, this patch just 'chmod -x' to all of them.

Closes-Bug: #1468564

Change-Id: Idec4ae16501d68e044e103db24ad3be0e0751e9a
2015-08-01 05:16:32 +00:00
Ihar Hrachyshka 22328baf1f Migrate to oslo.log
It's mostly a matter of changing imports to a new location.

Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
  from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
  would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
  not exist
* WritableLogger is now located in oslo_log.loggers

Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).

Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.

Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.

Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
2015-03-12 11:22:56 +01:00
Ihar Hrachyshka 1ae0a4b632 Moved several services into neutron.cmd.eventlet
- dhcp-agent
- l3-agent
- metadata-agent
- metadata-proxy
- metering-agent
- server

This allows us to remove explicit monkey_patch() call.

Also removed ability to execute neutron-server avoiding a corresponding
entry point.

Depends-On: I2d7081dbd4cb532332e3b66667bb8c71aa5a6658

Related-Bug: #1418541
Change-Id: I89e3e8e23374ab1a9a1844b3caaa88e162418546
2015-02-18 13:33:57 +01:00
Ihar Hrachyshka 7a2a85623d oslo: migrate to namespace-less import paths
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.

This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils

Added hacking check to enforce new import paths for all oslo libraries.

Updated setup.cfg entry points.

We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.

[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages

Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/

Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
2015-02-05 15:09:32 +01:00
Ihar Hrachyshka 3f44c9e278 Migrate to oslo.i18n
Mostly trivial import changes.

- oslo.i18n no longer provide install() method to inject _() into
  globals(), so removed all calls to it;
- removed Babel from dependencies (it will now be grabbed by oslo.i18n);
- updated tox.ini to ignore import violations for oslo.i18n.

Change-Id: I6623d551f512fb7fe9bf35ee734ed6d4c6cbc287
2014-11-26 22:19:24 +01:00
Gary Kotton 25d3e0114a Update i18n translation for neutron.server/scheduler log msg's
Validate that hacking rules apply to directories:
- neutron/scheduler
- neutron/server

Change-Id: I1353f5c8b01f85e4995ee1fb23a066506ed98e3f
Partial-bug: #1320867
2014-11-19 23:25:48 -08:00
Ihar Hrachyshka c0db9c23e3 Configure agents using neutron.common.config.init (formerly .parse)
After oslo.messaging port is done, we'll need to initialize RPC layer
from all RPC agents. We'll put initialization into init() function, so
the first step for migration is to make agents use it.

The function is renamed to be explicit about the fact that we don't just
parse configuration by calling it, but also do other common
initializations, like setting RPC layer.

blueprint oslo-messaging

Change-Id: I71c62f270ac7a1ff1426a3f49a32133b65580e35
2014-06-17 21:56:24 +02:00
Ihar Hrachyshka 6ca8cb84fd Monkey patch threading module as early as possible
In oslo.messaging, local thread store is used to pass RPC request
context [1]. If we try to import oslo.messaging before monkey patching
threading library, it attempts to access unpatched storage and fails
with AttributeError.

[1]: oslo/messaging/localcontext.py#L26

blueprint oslo-messaging

Change-Id: Ied7302fcb1d3e14428540e39e3db704550027890
2014-06-09 16:42:47 +02:00
Carl Baldwin d925b8fb9c Adds multiple RPC worker processes to neutron server
blueprint multiple-rpc-workers

Co-Authored-By: Terry Wilson<twilson@redhat.com>
Change-Id: I51f2a52add6c11af905e6f1e6e45d31731ebbb5d
2014-03-04 00:50:20 +00:00
armando-migliaccio 2d174a1876 Update License Headers to replace Nicira with VMware
In the process __init__.py involved have been emptied;
vim modelines and author lines have been dropped from
the license headers affected by the change as well.

Partial-implements blueprint nicira-plugin-renaming

Change-Id: I9c535b59bb2e83f275816472e0e32b41444d3bbe
2014-02-27 08:11:15 +00:00
Luis A. Garcia 5c36028a8c Re-enable lazy translation
After enhancements to Oslo Message class, re-enable lazy translation to
enable REST API responses to be translated to the requested user locale.

bp i18n-messages

Change-Id: I80272fb0d063a78882611b3044c920e28cc57d6b
2014-02-10 19:26:09 +00:00
Ben Nemec 76e7542f0f Disable lazy translation
Late in the Havana cycle bug 1225099 was found in the lazy
translation code, and to be safe it was decided to disable lazy
translation for Havana. This change does that.

Change-Id: Ia934a7df9386baf6ae8eb9ff48c24386c47ecd23
Partial-bug: 1225099
2013-10-01 23:16:23 +00:00
Luis A. Garcia d12fcadaa0 Enable localizable REST API responses via the Accept-Language header
Add support for doing language resolution for a request, based on the
Accept-Language HTTP header. Using the lazy gettext functionality, from
oslo gettextutils, it is possible to use the resolved language to
translate exception messages to the user requested language and
return that translation from the API.

Partially implements bp user-locale-api.

Change-Id: I63edc8463836bfff257daa8a2c66ed5d3a444254
2013-08-07 21:19:25 +00:00
Jason Dillaman 34b8ee5d59 Ensure that the neutron server is properly monkey patched
The legacy 'quantum-server' script will apply the eventlet
monkey patch whereas the new 'neutron-server' entry point
does not.  This prevents the server from properly handling
the REST interface.

Fixes bug #1199383

Change-Id: I498d2b02e6018425fc4538eff6803df8971f8e82
2013-07-09 14:08:21 -04:00
Mark McClain ee3fe4e836 Rename Quantum to Neutron
This change renames everything to Neutron while providing backwards
compatible adjustments for Grizzly configuration files.

implements blueprint: remove-use-of-quantum

Change-Id: Ie7d07ba7c89857e13d4ddc8f0e9b68de020a3d19
2013-07-06 15:02:43 -04:00