Sometimes Neutron is failing to send notification to Nova
due to timeout, refused connection or another HTTP error.
Retry send in those cases.
Closes-Bug: #1987780
Change-Id: Iaaccec770484234b704f70f3c144efac4d8ffba0
Nova notifier logs for external events only provide the event
details without showing its connection to the associated request_id.
That complicates debugging of corner cases for external events as we
cannot be sure to which exact request a logged event is related.
Make the matching request_ids logged alongside external event
details, whenever it is available.
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
Change-Id: I047a28744a0bf52ace9f9a205993c6037a21083c
The Nova Notifier can be called simultaneously by several RPC
callbacks from the agents (DHCP, L2), trying to update the provisioning
status of a port. In order to handle each context notifier enable flag,
a thread local variable is used. This will isolate the flag update
if two entities inform at the same time and one RPC callback is attended
during the processing of the other one.
This patch also removes the debug messages added to debug this issue.
Closes-Bug: #1958363
Change-Id: Ie670fba4b3afe427747732d2c3948d92311e960e
I should helps with debugging why the nova notifier is at some point
disabled for one neutron worker thus notifications aren't send to
nova from that worker.
Related-Bug: #1958363
Change-Id: Ifd88d5bc7515b5f29240bd12be1d3257b308fd67
The goal of this patch is to avoid the connection disruption during
the live-migration using OVS. Since [1], when a port is migrated,
both the source and the destination hosts are added to the profile
binding information. Initially, the source host binding is activated
and the destination is deactivated.
When the port is created in the destination host (created by Nova),
the port was not configured because the binding was not activated.
The binding (that means, all the OpenFlow rules) was done when Nova
sent the port activation. That happend when the VM was already
running in the destination host. If the OVS agent was loaded, the
port was bound seconds later to the port activation.
Instead, this patch enables the OpenFlow rule creation in the
destination host when the port is created.
Another problem are the "neutron-vif-plugged" events sent by Neutron
to Nova to inform about the port binding. Nova is expecting one single
event informing about the destination port binding. At this moment,
Nova considers the port is bound and ready to transmit data.
Several triggers were firing expectedly this event:
- When the port binding was updated, the port is set to down and then
up again, forcing this event.
- When the port binding was updated, first the binding is deleted and
then updated with the new information. That triggers in the source
host to set the port down and the up again, sending the event.
This patch removes those events, sending the "neutron-vif-plugged"
event only when the port is bound to the destination host (and as
commented before, this is happening now regardless of the binding
activation status).
This feature depends on [2]. If this Nova patch is not in place, Nova
will never plug the port in the destination host and Neutron won't be
able to send the vif-plugged event to Nova to finish the
live-migration process.
Because from Neutron cannot query Nova to know if this patch is in
place, a new temporary configuration option has been created to enable
this feature. The default value will be "False"; that means Neutron
will behave as before.
[1]https://bugs.launchpad.net/neutron/+bug/1580880
[2]https://review.opendev.org/c/openstack/nova/+/767368
Closes-Bug: #1901707
Change-Id: Iee323943ac66e566e5a5e92de1861832e86fc7fc
This option allows to configure Number of times nova or ironic client
should retry on any failed http call.
Default value for this new option is "3".
Change-Id: I795ee7ca729646be0411a1232bf218015c65010f
Closes-Bug: #1883712
Neutron makes call to nova for sending notifications but here no
context object user/admin available, so generated a request_id
and passed it as global_request_id during novaclient initialization
so that nova will log both global_request_id and it's own generated
request_id in the context formatter for traceability. As Notifier
class is used as singleton mode, need to create novaclient every time
events are sent from neutron to nova.
Modified unit tests wherever applicable.
Oslo spec I65de8261746b25d45e105394f4eeb95b9cb3bd42
Change-Id: I94257bd6ec9ec6b9a1f509c27c439e6305e43e63
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.
Trivialfix
Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
This patch switches callbacks over to the payload object style events
[1] for BEFORE_RESPONSE and AFTER_REQUEST based notifications. To do
so an APIEventPayload object is used with the publish() method to
pass along the API related data. In addition a few UTs are updated to
work with the changes.
NeutronLibImpact
[1] https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html#event-payloads
Change-Id: Ibd8559e0db9dcc995abf8937a0cb764b21a18531
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
While running unit tests plugins/ml2/drivers/test_type_vlan.py
the following message is observed:
""" UserWarning: Discovering extensions only by contrib path is
no longer supported since all contrib extensions have either
been made required or removed. The only_contrib argument is
deprecated and will be removed in a future release.
"""
From the source code you can see the TODO message to remove
the argument in Queens.
264c22a9cc/novaclient/client.py (L181)
This commit removes the argument and the stderr captured by unit
tests.
Change-Id: Ia4270c69c1519342636206fa439640240a1e13bb
When Nova gets a network-changed event, it rebuilds the
entire network info cache for the instance if it does not
have a specific port ID. This can be costly and redundant
when performing something like a live migration with multiple
ports attached to the same instance.
This change simply adds the port ID to the network-changed event
since we have it in scope. Nova can use it or not, but at least
the information is provided for context.
Change-Id: Ifdaef05208d09ddd9587fed6214cf388e5265ba4
Closes-Bug: #1691602
The callback modules have been available in neutron-lib since commit [1]
and are ready for consumption.
As the callback registry is implemented with a singleton manager
instance, sync complications can arise ensuring all consumers switch to
lib's implementation at the same time. Therefore this consumption has
been broken down:
1) Shim neutron's callbacks using lib's callback system and remove
existing neutron internals related to callbacks (devref, UTs, etc.).
2) Switch all neutron's callback imports over to neutron-lib's.
3) Have all sub-projects using callbacks move their imports over to use
neutron-lib's callbacks implementation.
4) Remove the callback shims in neutron-lib once sub-projects are moved
over to lib's callbacks.
5) Follow-on patches moving our existing uses of callbacks to the new
event payload model provided by neutron-lib.callback.events
This patch implements #2 from above, moving all neutron's callback
imports to use neutron-lib's callbacks.
There are also a few places in the UT code that still patch callbacks,
we can address those in step #4 which may need [2].
NeutronLibImpact
[1] fea8bb64ba7ff52632c2bd3e3298eaedf623ee4f
[2] I9966c90e3f90552b41ed84a68b19f3e540426432
Change-Id: I8dae56f0f5c009bdf3e8ebfa1b360756216ab886
Nova V2.1 API has been available since Kilo, and the API is CURRENT.
Neutron is using Nova API via novaclient and the used Nova API is
V2.1 on the gate as the default Nova endpoint. So this patch doesn't
change the used Nova API actually, but current novaclient skips checking
expected API version in a response from Nova API.
This patch will be helpful to avoid an issue when enabling this check
on novaclient side.
NOTE: Cinder is using Nova v2.1 API with the similar patch as
Iadd3363265be6c5a8ed46704a712da6c15e2b046
Change-Id: Ie3a10f7078f4213da77be522e3866f82b5e8683c
This is a revert of change I3d53bff8278dabafd929ecbea0b4b3b441c9e1cf
The nova notifier was updated to notify nova on ports with the
baremetal: device_owner, these ports are owned by Ironic not Nova, so
nova is getting notifications that it doesn't understand.
Change-Id: I8318a682163f6a5b739be68ce56973c43d0e32f2
Closes-Bug: #1656010
Depends-On: I43c3af9f424a65211ef5a39f13e4810072997339
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
By adding a singleton getter method and using it in the places
that use nova notifiers, we ensure that only one object exists
so all callers share the same batching queues and the sqlalchemy
events will only be fired once.
Change-Id: I89752d9c69feb578f0294339aae1a5cf51ec124b
Closes-Bug: #1622503
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.
Fix miscellaneous other deprecations.
Uses constants instead of l3_constants when importing
neutron-lib constants.
Co-Authored By: Henry Gessau <gessau@gmail.com>
Co-Authored By: Gary Kotton <gkotton@vmware.com>
Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
Use the only_contrib option that was added by
I030f4c55c2795c7f7973f5f12e54b9819c4a5578 to speed up nova_client calls
and skip the search for nova_client extensions to reduce
/var/log/audit noise.
Change-Id: Ic97b342a3633ffdf05b02ddd81baad88e1605a75
Closes-Bug: #1509500
Interface should be correctly removed from ironic when port-delete.
Nova should receive notify when baremetal port deleted.
Change-Id: I3d53bff8278dabafd929ecbea0b4b3b441c9e1cf
Partial-Bug: #1606229
If we try to delete a VM and to delete the floating IP
associated with the VM at the same time, depending
on the order according to which these requests are processed
Neutron might fail in the deletion of the floating IP,
raising a PortNotFound error. This happens because Neutron
notifies Nova of the network change event and it tries to get
the port to which the FIP is associated. If the port is not there,
Neutron shouldn't raise, it shouldn't send any notification.
Change-Id: Ic72313ad1f787b3cb528e806c843f1fd01eb12f2
Closes-bug: #1586931
- during live migration on pre migration step nova plugs instance
vif device on the destination compute node;
- L2 agent on destination host detects new device and requests device
info from server;
- server does not change port status since port is bound to another
host (source host);
- L2 agent processes device and sends update_device_up to server;
- again server does not update status as port is bound to another host;
Nova notifications are sent only in case port status change so in this case
no notifications are sent.
The fix is to explicitly notify nova if agent reports device up from a host
other than port's current host.
This is the fix on neutron side, the actual fix of the bug is on nova side:
change-id Ib1cb9c2f6eb2f5ce6280c685ae44a691665b4e98
Closes-Bug: #1414559
Change-Id: Ifa919a9076a3cc2696688af3feadf8d7fa9e6fc2
Once the decoupling started with 60c05a6d57, this cleanup
patch continues the effort by improving the decoupling
between the API layer and Nova.
Change-Id: I1d7d4b80ee77deefce18df22f76cab81750c0397
This patch adds a the Nova notifier calls to the pecan
notification hook and adds some simple functional tests
for them as well.
This patch also alters the operations of the policy enforcement
hook, which now stores in the request context the original
value of the object rather than the value of the object as it
would have appeared to the plugin - that is to say a merge
between the original objects and the parameters passed on to
the request body. Such 'frankenobject' is indeed of no use for
the notifiers.
Partial-Bug: #1552979
Change-Id: I4a37197eb38afe15d2b368f4e355226824478792
The NotFound response from Nova happens under normal circumstances, so
it should be a debug message, not a warning. In particular, if nova
itself deletes the port, when we then notify about it, it's going to say
not found.
Change-Id: I782a0fb9f39c2533bcf62b4e7ceb655f067c0282
Signed-off-by: Russell Bryant <rbryant@redhat.com>
When the neutron notification to nova was updated to use novaclient the
nova_url parameter was disabled. This prevents administrators from
using anything but the publicURL as the proper endpoint to notify nova.
This patch adds an option to pass on to novaclient for the
endpoint_type so that the administrator can set the notification url to
public, internal or admin.
Change-Id: I405f761944449cab6b8c8895f98419f79cd74cad
Closes-Bug: #1478471
DocImpact: Need to add a new option to the neutron configuration
reference.
keystoneauth was split out last cycle as a library specifically to deal
with doing auth functions so that people who do not need to do keystone
CRUD operations can just consume only the auth session parts. As part
of modernizing keystone interactions, use keystoneauth instead of
keystoneclient.
Depends-On: I1f754a9a949ef92f4e427a91bbd1b1e73e86c8c4
Change-Id: Ia6fb7d8bb07d3101cf1b1afc318f3dcac0a37afc
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
Now that we have the constant defined, we should reuse it from other
code to avoid potential typos.
Change-Id: Iebb270be46b116df3441370dc1a6784571311aa9
Neutron nova_* options have been deprecated in Kilo in favour of options
in [nova] section.
This change removes these options in order to simplify code, clarify
configuration and avoid a warning in unittests.
DocImpact
UpgradeImpact
Closes-Bug: #1509092
Related-Bug: #1403686
Change-Id: I950f72ad7d7d41cfa006f8c7105934301cd56faa
novaclient provides a common way to discover all extensions, so we can
remove import based on novaclient versioned client object.
Closes-Bug: #1493886
Change-Id: I7ae2eeb2d7e5c56e9284f3b059ff6e3545f42d5f
This reverts commit 6575db592c.
Depends-on: I998b6bb80cc0a81d665b61b8c4a424d7219c666f
DocImpact
If Neutron is upgraded to Liberty before the Nova API is,
the Nova API log will contain errors complaining that it doesn't
understand this new event. Nothing will be broken, but there will
be an error every time a port is deleted until Nova is upgraded.
Change-Id: I7aae44e62d2b1170bae31c3492148bfd516fb78b
We need to wait until the nova support is added in
I998b6bb80cc0a81d665b61b8c4a424d7219c666f. Otherwise
this generates a ton of error messages in the nova api
log as well as on the neutron side.
This reverts commit 0ace88fd4a.
Change-Id: I129c4e4c05cf07d45032fec6f57e0cc17a5a82af
Closes-Bug: #1461391
It's possible to delete a neutron port that is currently associated
with an instance. When it happens, neutron should notify nova of the
port deletion event so that Nova can take proper actions.
Refer to I998b6bb80cc0a81d665b61b8c4a424d7219c666f for the nova patch
that handles the event.
Change-Id: Iff88cd12ae18017ef3e776821bcf3ecf3b4f052f
Related-Bug: #1333365
Related-Bug: #1448148
Refactor the batch notifier currently used by the Nova notifier
into a separate class. It will be reused when batching L3 HA
state change events.
Partially-Implements: blueprint report-ha-router-master
Change-Id: I2f8cf261f48bdb632ac0bd643a337290b5297fce
It's mostly a matter of changing imports to a new location.
Non-obvious changes needed:
* pass overwrite= argument to oslo_context since oslo.log reads context
from its thread local store and not local.store from incubator
* don't store context at local.store now that there is no code that
would consume it
* LOG.deprecated() -> versionutils.report_deprecated_feature()
* dropped LOG.audit check from hacking rule since now the method does
not exist
* WritableLogger is now located in oslo_log.loggers
Dropped log module from the tree. Also dropped local module that is now
of no use (and obsolete, as per oslo team).
Added versionutils back to openstack-common.conf since now we use the
module directly from neutron code and not just as a dependency of some
other oslo-incubator module.
Note: tempest tests are expected to be broken now, so instead of fixing
all the oslo.log related issues for the subtree in this patch, I only
added TODOs with directions for later fix.
Closes-Bug: #1425013
Change-Id: I310e059a815377579de6bb2aa204de168e72571e
This allows a variety of additional authentication methods and gives us
keystone V3 auth for free.
DocImpact: Deprecates the nova_admin_* config options in favour of
putting nova config options in the [nova] section.
Closes-Bug: #1403686
Change-Id: Ia9941cd3e85f21cbd4db9eb3cef8b200131af9ce
novaclient has specific function novaclient.client.Client for obtaining
client object. This fuction should be used instead of direct import.
Also, contrib path dependends on version, so we should get it based on
versioned client.
Change-Id: If9c55446c4d10a58e9723f5c333082bcacb431b8
Closes-Bug: #1418017
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
Szymon Wroblewski