Neutron lib contains the latest callbacks and thus this patch removes
the callbacks package from neutron entirely.
NeutronLibImpact
Change-Id: I14e45fd5d2d3c816bb39f8ace56f7be460bac0d6
SUBNET_GATEWAY is not a real resource. It just corresponds to when
an attribute of the SUBNET changes. We can just supercede it entirely
by having everyone use the SUBNET resource, which solves significantly
more use cases.
Change-Id: Iae467bacb083888f72b8c9b52a10e4928a3ff6ab
Closes-Bug: #1690944
Add callback (PRCESS, AFTER_SPAWN) to do housekeeping task.
ML2 driver(in fact, networking-odl) wants to run house keeping task
which needs to run only in api server.
The following is very netwokring-odl specific as FYI:
The task is to run a timer which is extended when rest requests are
handled. The timer is fired to kick journal thread only when rest
request is idle in order to check if there is journal entries that was
created by other neutron servers that crash before processing them.
Such timer isn't needed for each workers. Only single timer among api
worker and rpc workers is enough.
cf https://review.openstack.org/#/c/461620/
The current approach is to run timer unconditionally for all processes
and stop it if it's worker process. With this patch, the timer can be
simply run for main process(api worker).
Change-Id: I9c07bc528c3a2fade0c835797889fc169f9bd1a6
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #1 from above, shimming neutron's callbacks and
removing devref + UTs. Rather than shimming using debtcollector, this
patch leaves callback constants as-is, and simply references the lib
class/function in its respective neutron callback module. This allows
consumers to test callback types without changing code. For example,
an except block block like that below continues to work even though
the raised exception now lives in lib::
try:
neutron_cb_registry.notify(...)
except neutron_cb_exceptions.CallbackFailure:
handle_exception()
In addition this patch contains minor UT updates to support the shim
approach.
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
Change-Id: Ib6baee2aaeb044aaba42a97b35900d75dd43021f
external networks are not really independent resources that
have their own lifecycles independent from networks. The only
use cases for this callback can be achieved with just regular
NETWORK callbacks.
The only users of this event have been adjusted to use NETWORK
callbacks (according to codesearch.openstack.org).
Closes-Bug: #1678041
Change-Id: Ie8249ddc3de643bfcb5c97480f30dc3500699256
These callbacks perform validation of user behaviors so
logging an ERROR isn't the right thing to do here. A debug
statement is all that is necessary since any unexpected exceptions
will be logged by the exception handler in the API layer.
TrivialFix
Change-Id: Iad671cd499d5fcb720702ba94aab061d04801fe7
This change updates has_registry_receivers implementation in order to
call super.__new__ without args, kwargs parameters if super.__new__ is
object.__new__ (which forbids args, kwargs parameters). Without this fix
decorated class instantiation with positional or/and keyword arguments
fail.
Change-Id: I25880f6142f294fdde9db7f453d5e649c79029d0
When a class doesn't define __new__, the __new__ method is inherited
from the parent class using MRO; however, we capture the orig_new at
class definition time which doesn't have the definitive MRO because
we don't know how the class will be used. So calling orig_new when
it's inherited ends up skipping sibling class __new__ methods.
This fixes the has_registry_receivers decorator to detect the case
when a class doesn't have a custom __new__ method and performs
a super() call like a custom __new__ method would to ensure we
don't miss calling __new__ on any classes in the heirarchy.
Change-Id: I547a887dac113625a1045d95bab099059e99fc0c
This adds a simple flag to object instances to avoid multiple calls
to 'subscribe' for a class with multiple children that use the
'has_registry_receivers' decorator.
The registry manager protects from multiple subscriptions of the
same function by storing callbacks based on IDs, but this is necessary
to eliminate the duplicate debug logging statements.
Change-Id: I38e0a5f4c361435ae4af6c16a407191b1e722e37
This adds some sugar to allow subscription to resources via
a decorator directly on a method as it is defined in a class.
This will allow us to stop overriding __new__ everywhere for
subscriptions and will allow the fact that a method is subscribed
to a resource to be visible right above the method itself.
See the follow-up patch[1] for an example of how this can clean
things up.
1. I764be99b2edbd0e7b54673c1940e883d7f313bf5
Change-Id: I1020c5ddec57f535e5b94f803144dfa19d668937
Logging the ID of the callback should be enough
since the ID contains the path to the callback.
Before:
Notify callbacks
[('neutron.services.segments.db._update_segment_host_mapping_for_agent-8784700722903',
<function _update_segment_host_mapping_for_agent at 0x7fd58f74ed70>)]
for agent, after_create _notify_loop"
After:
Notify callbacks
['neutron.services.segments.db._update_segment_host_mapping_for_agent-8784700722903']
for agent, after_create _notify_loop"
TrivialFix
Change-Id: I01d886f2707479183d7ae9c530f5f3b7b3963340
Using Nova's generic resource pools (GRP) API, publish routed networks
IPv4 inventory. This inventory is then used by the Nova scheduler to
place instances based on the availability of IPv4 addresses in
routed networks segments.
Change-Id: Ib6b00c4889d6a34765844ce46280819dff0108c5
Partially-Implements: blueprint routed-networks
This new event is aimed at informing that OVS has restarted, and in particular
to let L2 extensions know that they may need to setup their flows again.
Change-Id: I9aebe7ccc3e2f565b4339d42842d89b911131b1f
Closes-Bug: 1646526
Partial-Bug: 1657689
The tool has become popular beyond my wildest expectations. A typical
run in the gate at debug level yields more than 20K traces in the
neutron-server alone. However, reducing the number of traces is
beneficial across the board and that's the first attempt at bringing
the size of Neutron logs closer to other services'.
Partial-bug: #1620864
Change-Id: I2576e59961ec8928ad71d778253b2d37d470d909
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.
Fix miscellaneous other deprecations.
Uses constants instead of l3_constants when importing
neutron-lib constants.
Co-Authored By: Henry Gessau <gessau@gmail.com>
Co-Authored By: Gary Kotton <gkotton@vmware.com>
Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
The callback manager was indexing callbacks based on
a callback ID generated from oslo utils reflection.
This presented two problems.
The first was that in py34 get_callable_name would use
__qualname__ which returns the class name the function
is defined on rather than the class of the object itself.
So two classes defined in the same module inheriting from
the same parent class could not both subscribe a method
defined on the parent.
The second more general problem is that two objects which
are instances of the same class cannot subscribe the same
method because they have the same ID.
This adds the hash of the method to the ID to prevent these
issues. The hash by itself could have been used but it's not
very user-friendly so the name is left on for nice log
messages.
Change-Id: Iff1ca8c4ddb58ca5907d21fa0de7f0f292b6fc0e
Some of the agent code may want to receive events whenever
a device is added/updated/deleted. Currently the only way
to do that is to modify the loop directly or create an
extension. Neither of these options are good if it's code
we don't want users to disable (the extensions) and it's
specific to one agent (so we don't want to modify the loop).
Partially-Implements: blueprint vlan-aware-vms
Change-Id: I3468c7f46cc1b4000cdd537e8f216d207a14727a
The current PROCESS AFTER_INIT event (or AFTER_CREATE) are being called
multiple times, once for each process.
In case we want to perform an action only once after init, we would need a
callback that is called after init is done, but before spawning.
Usage examples can be:
- A plugin/driver needs to configure something at the backend after init is done
- A plugin/driver needs to create an object in one of the neutron tables
(add a network for example)
Change-Id: I249ba32dede16e61b1c757a8010dac610484da4f
Change 877778ee4c clarified the objective of the callback
registry events, and it introduced two new API related events.
This change introduces a new set of events that are meant
to be used for handling process-related operations, in fact
even though AFTER_CREATE was fit for the job, it looks somewhat
misused now.
Change-Id: I9e717e9e4ae4414067477712ac02870fe1041e70
This adds logic to add segment to host mappings for agents
after a segment is created. The previous logic was doing this
only after an agent initially reported after the server started.
This meant that any segments created after the agent was running
would not get mappings to the agent's host.
Partially-Implements: blueprint routed-networks
Change-Id: Ibcd9964afb1c7ac3926447f90b3fd364fd4835ca
Bug 1591766 unveiled an issue where calling the plugin API does not trigger
DHCP notifications. This is required by the auto-allocated-topology service
plugin that calls core_plugin.update_network(), and expect notifications
to be sent out on state changes. To accomplish this, the logic has been
encapsulated in the DHCP module, and leveraged via callback mechanisms.
For this reason, new events have been introduced, AFTER_REQUEST, and
BEFORE_RESPONSE. The latter in particular is the one needed to hook up
dhcp notifications in order to preserve backward compatibility.
More precisely, core plugins that use DHCP as is or implement their own,
(with or without an agent) should already instantiate their own notifier,
and if they do not, this should be rectified.
A search on codesearch.openstack.org reveals that out-of-tree plugins
already specify their own notifiers, and the default initialization is
clearly redundant now.
Related-bug: #1591766
Change-Id: I7440becb6d30af7159ecaeba09d7a28eceb71bea
There are various places where we emit a callback to allow
subscribers to do validation or additional bookkeeping during
the various life-cycle events of objects. However, a subscriber
can encounter a DB Deadlock, StaleDataError or other retriable
error that just needs the transaction to be restarted. In the
rest of the Neutron code these exceptions bubble up to the API
layer where the whole request is restarted. However, the exception
catching for the callbacks in various places was preventing these
errors from being retried because they were lost in conversion.
This patch has the callback manager convert retriable exceptions
into RetryRequest exceptions that will not be caught by handlers
of CallbackError so by default it will bubble up to the API layer
and be retried. 'notify' callers can avoid this behavior by
additionally catching RetryRequest.
Closes-Bug: #1590316
Change-Id: I6732c60f89de4318b5f56327c5bc966bd250baae
This adjusts the notify loop logic to handle the case where
a callback causes a subscription or unsubscription that changes
the subscriber dictionary to change during iteration.
It was just using .items() which solved the problem for py27 but
was not creating an actual copy in py34. This just calls list()
on .items() to make sure we get a list in both cases.
Change-Id: Iee9d675faf30ec714b4f5c77128d8843d545ecfd
Closes-Bug: #1592983
The callback framework will collect all exceptions that occur
during the notification loop and raise a single exception
containing all of them. The issue with this is that the code
that notifies has to manually unpack the exceptions and choose
what to reraise for the API layer even if it has no other reason
to catch the exception (i.e. any encountered exceptions should be
fatal). If it doesn't, the server will just return a generic HTTP
500 error to the user even if the internal exception is a normal
error that would convert to a 404, 409, etc.
This patch makes the API exception conversion layer aware of
exceptions containing other exceptions so code no longer has to
catch callback failures if it has no specific error-handling logic.
Multiple exceptions that translate to the same HTTP error code will
be converted into one exception of the same error code with the
details of each exception line-separated in the exception message.
Multiple exceptions that translate to different HTTP error codes will
be concatenated together line-separated with their HTTP error prefixes
into an HTTP Conflict exception.
If there is only a single exception in the multi exception type, the
inner exception is used directly.
Partially-Implements: bp/multi-l3-backends
Change-Id: I528de088079b68cf284ef361fee9bd195125e0d8
Once all agents are deleted from a host, the DVR MAC generated
for that host should be deleted as well to prevent a buildup of
pointless flows generated in the OVS agent for hosts that don't
exist.
Closes-Bug: #1568206
Change-Id: I51e736aa0431980a595ecf810f148ca62d990d20
Current BEFORE_CREATE/DELETE/UPDATE event is outside of the DB
transaction. Unlike the precommit primitive in ML2 mech drivers, they
don't work in the same DB transaction of resource, so if we want to
operate the DB in mech driver related to security group, there would be
more unsync issues if we use BEFORE_XXX event directly. Moving the
BEFORE_XXX event inside may also break some current codes, as maybe RPC
call included.
This patch adds new PRECOMMIT_CREATE/DELETE/UPDATE event type for
callback function, and use it in the securitygroup/rules DB transaction.
PRECOMMIT_XXX is in the DB transaction and only purpose is to do DB
operations in its callback. A CallbackFailure will be triggered when
exception comes from the callback of the new event.
Change-Id: Icd2849bd84dab6733a572e8c85f242afcefc6c78
Closes-Bug: #1522172
This patch introduces an API to allocate an externally connected
private tenant network on demand. The API is idempotent in that,
once the topology is provisioned, further API calls keep returning
the same topology to the caller.
The API, as introduced by the patch, is not currently on, and its
design carefully ensures minimal impact on the existing codebase.
In fact the feature depends on and enhances the external-net extension,
but it does so via callbacks.
A subsequent patch in this series will make it available by default,
and API tests will be added to validate the functionality.
Partially-implements: blueprint get-me-a-network
Co-Authored-By: Armando Migliaccio <armamig@gmail.com>
Co-Authored-By: Henry Gessau <HenryG@gessau.net>
Change-Id: I4abd45252026431452f0d2cb2805043489c2f6ad
When the address scope of a subnetpool changes, the scope of related
subnet will change too. If this subnet has been connected to a router,
the related routing information needs to be updated.
This patch will fire the router update to correlative routers. The
routers will get all things updated in the update action.
Change-Id: Ia2eb484eb949a8d42fc909291de7be07b002adda
Partially-Implements: blueprint address-scopes
This patch makes the L3 service generate notifications that enable
BGP dynamic routing to react appropriately to user-initiated changes
that affect prefixes that should be advertised and the corresponding
next-hops.
Partially-Implements: blueprint bgp-dynamic-routing
Co-Authored-By: vikram.choudhary <vikram.choudhary@huawei.com>
Change-Id: I96f14a31876efeca5881d97771e3263525522e36
The callback mechanism allows notifiers to provide keyword args in
the notification. If this is a create callback, and there is an
exception, then the notifier will call an abort callback.
However, currently, the keyword arguments are not provided to the
abort callback. This information could be useful for the callbacks,
and would make the mechanism consistent. This commit provides that
information.
Change-Id: I2ee0363b52f9de5fcd72905957e8f7e7796f630e
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
There are several cases where plugin initialization should be
handled after neutron-server forks API/RPC workers. For example,
starting a client connection to an SDN controller before forking
copies the fd of the socket to the child process, but then you have
multiple processes trying to read/write the same socket connection.
It is also useful for a plugin to be able to do something in only
one process, regardless of how many workers are forked. One example
would be handling syncing from an external system to the neutron
database.
This patch does 3 things:
1) Treats rpc_workers=0 as = 1. This simplifies the code for
handling notification that forking has completed. In the
existing code, calling the notification in the Worker object's
start() method would happen twice in the case where both api
and rpc workers were 0, despite there being only one process.
An earlier patch already changed the default api_workers to be
the number of processors.
2) Adds notification of forking via the callbacks mechanism.
Plugins can subscribe to resources.PROCESS, event.AFTER_CREATE
and do any post-fork initialization that needs to be done for
every spawned process.
3) Adds core/service plugin calls to get_workers() which defaults
to returning (). Plugins that need additional processes to spawn
should just return an iterable of NeutronWorkers that will be
spawned in their own process.
DocImpact
Closes-Bug: #1463129
Change-Id: Ib99954678c2b4f32f486b537979d446aafbea07b
This patch enables users to update gateway_ip of a subnet even if
the subnet is in use for an external network of a router.
Change-Id: I78d2b024c99b1af0001bd454465d2fc02692cbf2
Closes-Bug: #1317363
Latest developments have revealed that the registry can be misused
under certain circumstances, and that it can be harder to use by
projects that extend Neutron.
This patch improves the devref documentation so that developers know
what to expect.
Change-Id: I565b6a2f2a58bf22eae5b36f03c4fd24ba0774d2
When using LBaaS and trying to delete a subnet, neutron has no way of
knowing if the subnet is associated to some pool. As a result, the
subnet is deleted but the pool remains associated to the (now
nonexistent) subnet_id. This patch lays the ground-work for adding a
check in LBaaS' side to prevent such cases.
Related-Bug: #1413817
Change-Id: I3d5e231b67c72ffd919c92d65b57da56c63e053c
The use of weakref was introduced as a preventive measure to avoid
potential OOM kills, however that limited our ability to employ
certain functions as callbacks, such as object methods (see [1] for
an example).
Since the adoption of the callback registry, it has been observed that
callbacks are generally long lived (for the entire duration of the
process they belong to), therefore this limitation appears to be too
restrictive at this point in time.
Some might argue that it's better safe than sorry, but until we
have some evidence of actual OOM kills, it's probably best to take
the bolder action of removing the adoption of weak references and
deal with the potential fallout, should it happen.
[1] https://review.openstack.org/#/c/175179/
Change-Id: Idcd0286fc4235af82901c8a17ea45bc758b62b37
Boden R