With python 3.x, classes can use the metaclass= logic
to not require usage of the six library.
One step in removing all of six usage from neutron.
Change-Id: I2f815e412d9a96eb5faf2b3bb3a1e393a9db9309
The minimum value for dhcp_agents_per_network was being
inforced with an explicit check in code but we can do that
with simply defining the min value in the option itself.
This adds the minimum to the option definition and removes
the explicit check in the code. Note that as a result the
validate_post_plugin_load method needed to be removed
otherwise it fails the E1128 pylint check.
Change-Id: I103855d363c49fad4119ac850aed0a166d8c046d
In patch [1] requirement that only each service plugin
can be loaded only once was removed.
Unfortunatelly it is not possible that same service plugin
will be instantiate more than once because it may reqister some
callbacks or other things which can't be duplicated.
So this patch adds mechanism which will ensure that each service
plugin class is instantiate only once and reused if necessary.
[1] https://review.openstack.org/#/c/626561/
Closes-Bug: #1816771
Change-Id: Ie6e6cc1bbbe50ff7cfad4e8033e48711569ea020
Adds a required list 'required_service_plugins' to each service plugin,
then we can initialize the service plugin with required dependency.
And also adds the 'router' plugin to port forwarding service plugin
required list.
Closes-Bug: #1809238
Change-Id: I53fdaee0cd96a5315a7abc39799657d613eb3a2e
Enforce validation on filter parameters on list requests.
If an API request contains an unknown or unsupported parameter,
the server will return a 400 response instead of silently ignoring
the invalid input.
In resource attributes map, all filter parameters are annotated by
the ``is_filter`` keyword. Attributes with is_filter set to True
are candidates for validation.
Enabling filter validation requires support from core plugin and
all service plugins so each plugin need to indicate if it supports
the validation by setting ``__filter_validation_support`` to True.
If this field is not set, the default is False and validation is
turned off. Right now, the ML2 plugin and all the in-tree service
plugin support filter validation. Out-of-tree plugins will have
filter validation disabled by default.
An API extension is introduced to allow API users to discover this
new API behavior. This feature can be disabled by cloud operators
if they choose to do that. If it is disabled, the extension won't
be presented.
Depends-On: Ic3ab5b3ffdc378d570678b9c967cb42b0c7a8a9b
Depends-On: I4397df1c35463a8b532afdc9c5d28b37224a37b4
Depends-On: I3f2e6e861adaeef81a1a5819a57b28f5c6281d80
Depends-On: I1189bc9a50308df5c7e18c329f3a1262c90b9e12
Depends-On: I057cd917628c77dd20c0ff7747936c3fec7b4844
Depends-On: I0b24a304cc3466a2c05426cdbb6f9d99f1797edd
Change-Id: I21bf8a752813802822fd9966dda6ab3b6c4abfdc
Partial-Bug: #1749820
Some service plugins make the assumption that Neutron is running
with a datastore (e.g. revision and timestamps). As the datastore
setup is a responsibility of the Neutron core plugin, checking
that this is indeed true avoids errors for those plugins that do
not implement any DB backend (e.g. monolithic OpenContrail plugin).
Change-Id: I872fa6e3c3925e521150d79bba864101d9a5f648
Closes-bug: #1700651
neutron-lib now contains the load_class_by_alias_or_classname function.
This patch removes the function from neutron and uses it from lib.
NeutronLibImpact
Change-Id: I47ef642fcb3c8ded5eb7ed0a637033c7bcb3632b
neutron-lib contains the synchronized lockutils decorator as well as
the SYNCHRONIZED_PREFIX global. This patch consumes them from
neutron-lib and removes them from neutron.
NeutronLibImpact
Change-Id: I729da348e340509f2d09f8a6436716e2398f1583
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
In reviews we usually check import grouping but it is boring.
By using flake8-import-order plugin, we can avoid this.
It enforces loose checking so it sounds good to use it.
This flake8 plugin is already used in tempest.
Note that flake8-import-order version is pinned to avoid unexpected
breakage of pep8 job.
Setup for unit tests of hacking rules is tweaked to disable
flake8-import-order checks. This extension assumes an actual file exists
and causes hacking rule unit tests.
Change-Id: Ib51bd97dc4394ef2b46d4dbb7fb36a9aa9f8fe3d
The well known service type constants are in
neutron_lib.plugins.constants, but for legacy reasons a few still exist
and are referenced from neutron_lib.constants that we'd like to remove.
This patch switches references over to neutron_lib's plugin constants.
Change-Id: I1861448cec303725b30cef8f42029f467f9e03a3
Probably the most common format for documenting arguments is reST field
lists [1]. This change updates some docstrings to comply with the field
lists syntax.
[1] http://sphinx-doc.org/domains.html#info-field-lists
Change-Id: I8fb91bcb8ae4bfbfa68ddf7c38a4f1f5cf548b36
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Before the patch, we were silently swallowing the original ImportError
raised by load_class_by_alias_or_classname that could contain valuable
details about the reason of failure.
This patch reraises the original exception instead of replacing it with
the one with no context. It still logs an error before doing it to make
sure the operator will see it irrespective of what the caller will do
with the exception.
Also, a note to fellow reviewer: No, I don't think it needs a unit test
case. Do you?
Change-Id: Ib47b67da1e7e4b749abb8b1b93888e23d00da00a
The legacy controllers used their own references to the resource
attribute info while pecan was looking for it in the
attributes.RESOURCE_ATTRIBUTE_MAP. This aligns pecan controllers
with that of the legacy controllers, in that the pecan controllers
now store the their own resource attribute info.
This also fixes a bug that was unnoticed until the above change was
made. The ItemController's index method was not passing the *args and
**kwargs to the get method.
Change-Id: I1cc85daecfb19f73092e52678fd19ff1a2912195
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
1) It checks that person who wants to trace is trusted and knows
secret HMAC key.
2) It starts tracing in case of proper trace headers
and adds first wsgi trace point, with info about HTTP request
* Add initialization of osprofiler at start of service
Currently that includes oslo.messaging notifer instance creation
to send Ceilometer backend notifications.
Neutron client change: Ic11796889075b2a0e589b70398fc4d4ed6f3ef7c
Co-authored-by: Ryan Moats <rmoats@us.ibm.com>
Depends-On: I5102eb46a7a377eca31375a0d64951ba1fdd035d
Closes-Bug: #1335640
DocImpact Add devref and operator documentation on how to use this
APIImpact
Change-Id: I7fa2ad57dc5763ce72cba6945ebcadef2188e8bd
Previously the pecan controller creation was based on the
attributes.RESOURCE_ATTRIBUTE_MAP's keys after it had been fully
populated by all active extensions. The problem with the problem
with this strategy is that those keys did not necessarily define
the actual URI path, and if any extensions ever had subresources
(.i.e. /qos/policies/{policy_id}/bandwidth_limit_rules), the
attributes.RESOURCE_ATTRIBUTE_MAP was never populated with those.
Getting those subresources would require some very unsightly code
and strategies.
What this does is it grabs the resources that every extension
returns in the get_resources method. The get_resources method
returns a list of ResourceExtensions that do infact define the
URI, and does have a clear linkage to subresources. With this
in mind, the pecan initializer can now translate what get_resources
returns into pecan controllers with the correct paths.
The subresource work still needs to be done and will be done in a
follow-up patch. Also, there is an issue with the
auto-allocate-topology tests, but this too can be solved in
another patch so this review does not get bloated.
Change-Id: I3b42b4b603af97538f08b401d2740d3049276a43
This is needed for extensions like security-groups as resource
names do not use dashes but underscores.
Change-Id: I952abd261a3cf8ac964463dd50377905f08830c8
Closes-Bug: #1537929
Service plugins are a great way of adding functionality in a
cohesive way. Some plugins (e.g. network ip availability or
auto_allocate) extend the capabilities of the Neutron server
by being completely orthogonal to the core plugin, and yet may
be considered an integral part of functionality available in
any Neutron deployment. For this reason, it makes sense to
include them seamlessly in the service plugin loading process.
This patch, in particular, introduces the 'auto_allocate' service
plugin for default loading, as we'd want this feature to be enabled
for Nova to use irrespective of the chosen underlying core plugin.
The feature requires subnetpools, external_net and router, while
the first is part of the core, the others can be plugin specific
and they must be explicitly advertised. That said, they all are
features that any deployment can hardly live without.
DocImpact: The "get-me-a-network" feature simplifies the process
for launching an instance with basic network connectivity (via an
externally connected private tenant network).
Once leveraged by Nova, a tenant/admin is no longer required to
provision networking resources ahead of boot process in order to
successfully launch an instance.
Implements: blueprint get-me-a-network
Change-Id: Ia35e8a946bf0ac0bb085cde46b675d17b0bb2f51
The pecan controllers were not parsing out the case when a service
plugin is being used that changes the URI to
/v2.0/service_plugin/resource
This will check to see if the first resource after v2.0 is an
extension to a service plugin, and if it is it'll check the next
resource for a controller.
Change-Id: I9b6bd7afbbe91f1c8f0c1835b320dc41bfccff3f
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
Make flavor service profile store actual driver instead of
hardcoded dummy driver. Ensure service type on flavor persisted.
Raise ServiceProfileDriverNotFound if non-empty driver is not part
of ServiceTypeManager providers.
Raise ServiceProfileEmpty if profile has neither a driver nor
any metainfo.
Raise InvalidFlavorServiceType if invalid service type passed.
Show flavors associated with a profile, not just profiles associated
with a flavor, to ease diagnosis when ServiceProfileInUse raised.
Create method to extract provider given a flavor for use with
neutron-lbaas plus tests.
Ensure various boolean forms accepted for enabled flag.
To enable in DevStack, add to local.conf:
enable_plugin neutron https://git.openstack.org/openstack/neutron
enable_service q-flavors
Add associated unit tests. Fix tempest api test that used invalid
LOADBALANCERS service type.
Change-Id: I5c22ab655a8e2a2e586c10eae9de9b72db49755f
Implements: blueprint neutron-flavor-framework
Changed the interface_driver configure for agents from class
imports to stevedor aliases. The loading method needed to be
updated to load as a DriverManager. Backward compatability
for configuration as class import.
DocImpact
Change-Id: Ic349691989484286cd7c60eaf3ad1454c5852d1f
Closes-Bug: #1504536
1. Scope mock of 'open' to module
By mocking 'open' at the module level, we can avoid affecting
'open' calls from other modules.
2. Stop using LOG.exception in contexts with no sys.exc_info set
Python 3.4 logger fills in record.exc_info with sys.exc_info() result
[1], and then it uses it to determine the current exception [2] to
append to the log message. Since there is no exception, exc_info[1] is
None, and we get AttributeError inside traceback module.
It's actually a bug in Python interpreter that it attempt to access the
attribute when there is no exception. It turns out that it's fixed in
latest master of cPython [3] (the intent of the patch does not seem
relevant, but it removes the offending code while reshuffling the code).
Note that now cPython correctly checks the exception value before
accessing its attributes [4].
The patch in cPython that resulted in the failure is [5] and is present
since initial Python 3k releases.
The patch in fixtures that broke us is [6].
[1]: https://hg.python.org/cpython/file/tip/Lib/logging/__init__.py#l1412
[2]: https://hg.python.org/cpython/file/tip/Lib/logging/__init__.py#l575
[3]: https://hg.python.org/cpython/rev/73afda5a4e4c
[4]: https://hg.python.org/cpython/rev/73afda5a4e4c#l6.484
[5]: https://hg.python.org/cpython/rev/2ee09afee126
[6]: 67dd295694
Closes-Bug: #1503847
Closes-Bug: #1504053
Co-Authored-By: Ihar Hrachyshka <ihrachys@redhat.com>
Change-Id: I456b7846b8a53e4d3f8c91583685e0e1eaa84719
Pecan defines several efficient mechanism for routing requests to
the appropriate controller, but the current code for Neutron's
Pecan WSGI server basically uses Pecan hooks to route requests.
This patch partially fixes that, removing the 'resource_identifier'
pecan hook and replacing it with explicit pecan routes between
controllers added at resource registration time.
All the remaining hooks, like attribute_population and
policy_enforments, which were relying on finding the resource
name in the pecan.request threadlocal variable have been updated.
This patch also:
- ensures the appropriate plugin is always selected for a given
resource
- add a common NeutronPecanController base class for the classes
CollectionsController and ItemaController
- Fixes the way in which plurals and singulars are handled in
neutron.api.v2.resource_heper
Change-Id: I4ec0d2276c3974117b497228d289c3fb0dc5a140
There are several cases where plugin initialization should be
handled after neutron-server forks API/RPC workers. For example,
starting a client connection to an SDN controller before forking
copies the fd of the socket to the child process, but then you have
multiple processes trying to read/write the same socket connection.
It is also useful for a plugin to be able to do something in only
one process, regardless of how many workers are forked. One example
would be handling syncing from an external system to the neutron
database.
This patch does 3 things:
1) Treats rpc_workers=0 as = 1. This simplifies the code for
handling notification that forking has completed. In the
existing code, calling the notification in the Worker object's
start() method would happen twice in the case where both api
and rpc workers were 0, despite there being only one process.
An earlier patch already changed the default api_workers to be
the number of processors.
2) Adds notification of forking via the callbacks mechanism.
Plugins can subscribe to resources.PROCESS, event.AFTER_CREATE
and do any post-fork initialization that needs to be done for
every spawned process.
3) Adds core/service plugin calls to get_workers() which defaults
to returning (). Plugins that need additional processes to spawn
should just return an iterable of NeutronWorkers that will be
spawned in their own process.
DocImpact
Closes-Bug: #1463129
Change-Id: Ib99954678c2b4f32f486b537979d446aafbea07b
This patch introduces API and DB plugin for flavor framework.
API adds Flavors and Service Profiles which are resources
available only for admins to operate.
This framework then should be leveraged by advanced services.
Included tempest API tests in neutron tree
Implements: blueprint neutron-flavor-framework
Change-Id: I99ba0ce520ae3d8696eca5c994777c7d5ba3d4b1
Co-Authored-By: Doug Wiegley <dougw@a10networks.com>
Co-Authored-By: Madhusudhan Kandadai <madhusudhan.kandadai@hp.com>
oslo.service has graduated, so neutron should consume it.
Closes-Bug: #1466851
Depends-On: Ie0fd63f969f954029c3c3cf31337fbe38f59331a
Depends-On: I2093b37d411df9a26958fa50ff523c258bbe06ec
Depends-On: I4823d344878fc97e66ddd8fdae25c13a34dede40
Change-Id: I0155b3d8b72f6d031bf6f855488f80acebfc25d4
This also adds a check to neutron/hacking/checks.py that should catch this
error in the future.
Blueprint: neutron-python3
Change-Id: Ie7b833ffa173772d39b85ee3ecaddace18e1274f
IPAM Driver is loaded based on value of 'ipam_driver'.
Added new variable 'ipam_driver' in config.
DocImpact
Partially-Implements: blueprint neutron-ipam
Change-Id: Ia52ad70ef4f0b02cf82cfefcf50b9f1e30b05b79
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
Patch implements translation from class paths to neutron to class paths
to neutron_<adv_service>. It's achieved by defining entry point in
setup.cfg which is translated by stevedore.
There will be needed patches in advanced services tree calling
get_provider_driver_class() function before importing class.
This patch specifically fixes loading service plugins and
drivers for service plugin. Patches for agents are still needed in
neutron repo and adv services repos.
Alternative and better solution would be implementing new DriverType
to oslo.config, which will have callback to
get_provider_driver_class()-like function.
Co-Authored-By: Ihar Hrachyshka <ihrachys@redhat.com>
Change-Id: I76af175c4387326a4e5ff95c2f15d8b866dedab3
Partial-Bug: 1401895
This change enforces log hints use and removes debug level log
translation, modifications are validated through a hacking rule
and the change respects logging guidelines.
Change-Id: Ia4e6659b8b59732d88cd603b0e6c630fad37aae5
Closes-bug: #1320867
Convert several places in the code that were using the RpcCallback
compatibility class. This class eased the transition to
oslo.messaging. This patch drops the usage of this class in favor of
direct usage of the appropriate oslo.messaging API.
Part of blueprint drop-rpc-compat.
Change-Id: I955958e836635d43dabc7b39d050c1418b18cf8f
Most of this code will probably stay with us for quite some time, so
let's make things easier and consider them as our way of doing RPC.
blueprint oslo-messaging
Change-Id: Iaf353b23f9c54b82d1e02a6bd5a5960cec827c88
The method _load_service_plugins() in /neutron/neutron/managers.py
throw a ValueError using ',' to connect log and variables.
ValueError do not support using ',' to connect log and variables,
we should use '%' instead of ','
Change-Id: Ic9b3756ac2bc36ddc689a66c078e205e0674f513
Closes-Bug:#1264210
This class will be used to create proper self.target with appropriate
API version once we migrate to oslo.messaging.
blueprint oslo-messaging
Change-Id: I1fb5eb0aaac0d115fd84630e58b333e695ad4f5f
The unit tests were previously consuming an excessive amount of memory
(4GB+) due to plugin instances persisting in memory. Deallocation was
not possible where a combination of circular references and mocking
was involved. This patch ensures that only NeutronManager holds a
plugin reference and that all other references are instances of
weakref.proxy. Residual memory footprint for tox executed on a
12-core machine has been reduced to ~1.3GB. Plugin deallocation is
validated at the end of each test to prevent regressions.
This change also includes fixes to unit tests that depended on plugin
instances persisting across tests.
Partial-Bug: #1234857
Change-Id: Ia1f868c2d206eb72ef77d290d054f3c48ab58c94
Brian Haley