In I3e702b99fd5084e8090f93c384aa1f704edceaff, the event listening
features which handle expiration of relationships was "rehomed"
to neutron-lib. In fact it was duplicated, so that these events
run twice. This patch removes the events from neutron as it is
a very bad idea for this much complexity to be running twice.
Change-Id: I8d0321487e4b406ead7a237156ef2f221fb512b9
The context_manager is already in neutron-lib and being used by our
active consumers either directly or via the current shim we have
in neutron.db.api.context_manager that just references that in lib.
This patch removes the context_manager from neutron and updates the UTs
that mock it.
NeutronLibImpact
Change-Id: Ia5191a71b377e2862592a09da8dc3a5d415de9aa
sqlalchemy osprofiler is set up for Neutron by neutron-lib in [1]. As a
consequence, there is no need to set it up in Neutron. Besides, when
profiling Neutron code, this duplication creates repetitive output in
the reports. This patch removes this set up from the DB API module.
[1] https://github.com/openstack/neutron-lib/blob/master/neutron_lib/db/api.py
Change-Id: I15f3ee374333e3cba2bf659439dd82263681f1df
The MAX_RETRIES constant from neutron.db.api is available in neutron-lib
and is already used by most of the code today. This patch changes the
remaining references over to use lib's constant and removes the
MAX_RETRIES from neutron.db.api
NeutronLibImpact
Change-Id: I79480daca9cca3d4e1504767d286aab22504e0b9
There are a handful of functions in neutron.db.api that are no longer
used due to the consumption of the db api from neutron-lib. This patch
removes them.
Removed:
- _copy_if_lds()
- add_to_rel_load_list()
- load_one_to_manys()
- LOG
Renamed:
- expire_for_fk_change is now named _expire_for_fk_change to indicate
it's private and should not be used by consumers.
NeutronLibImpact
Change-Id: Icbdad6712345d04b0b45194422cbe0813cd7e1e4
The retry_if_session_inactive decorator was rehomed into neutron-lib
[1]. This patch consumes it by removing the function from neutron and
using neutron-libs version where appropriate.
NeutronLibImpact
[1] https://review.openstack.org/#/c/557040/
Change-Id: I3e3289f33e62d45933d0fbf165bb4b25078f22d5
The db api is_retriable function is now available in neutron-lib. This
patch consumes it by removing is_retriable and related internal
functions and using neutron-lib's version where applicable.
NeutronLibImpact
Change-Id: Ie33a4eb4d1e52b2d985c9498edea2efb63ef569c
A publically consumed db api is available in neutron-lib. This patch
consumes the autonested_transaction by removing it's def from neutron
and replacing all such usages to now use neutron-lib's version of
autonested_transaction.
NeutronLibImpact
Change-Id: I8d7cbb9b90151d252959e9fce8937f3e4e456811
The public db apis are available in neutron-lib. This patch consumes
the sqla_listen, sqla_remove and sqla_remove_all functions from lib
by removing them from neutron and using lib's version.
There's also a TODO added here so we can eventually rehome the
check_no_sqlalchemy_event_import hacking check to neutron-lib.
NeutronLibImpact
Change-Id: I3b3862016125303fb227b4ff41420a89bb33c917
The externally consumed APIs from neutron.db.api were rehomed into
neutron-lib with https://review.openstack.org/#/c/557040/
This patch consumes the retry_db_errors function from lib by:
- Removing retry_db_errors from neutron.db.api
- Updating the imports for retry_db_errors to use it from lib
- Using the DB API retry fixture from lib in the UTs where applicable
- Removing the UTs for neutron.db.api as they are now covered in lib
NeutronLibImpact
Change-Id: I1feb842d3e0e92c945efb01ece29856335a398fe
The get_reader_session and get_writer_session functions are available
in neutron-lib. This patch consumes them by removing the functions
from neutron and using lib's instead.
NeutronLibImpact
Change-Id: I4055172799a0d9a0ceb387b6d47e421e1c270ada
This incorporates flake8 2.6.x and pycodestyle will be used
instead of older pep8. This ensures future python3 compatibility
and a bit better code styling.
Change-Id: Ia7c7c5a44727f615a151e1e68dd94c7ed42f974f
The publicly used APIs from neutron.db.api were rehomed into neutron-lib
with https://review.openstack.org/#/c/557040/
This patch consumes exc_to_retry from neutron-lib by removing it in
neutron and using it from lib.
NeutronLibImpact
Change-Id: I785c20f0df7a281c135e8e3fc8af439e6e481d52
This reverts commit a75014792a.
This is the second attempt to merge the patch after the previous one resulted
in revert due to multiple gate breakages in dependent projects (neutron-lbaas,
vmware-nsx, heat, networking-odl). This second attempt is validated with a set
of depends-on patches for all projects that were affected during the first
failed attempt.
The original commit message for the patch is included below for context.
===
Listen for foreign key changes and expire related relationships.
With this, we can remove OVO code that refreshes / detaches models on
each fetch. The patch also removes a bunch of expunge calls in plugin
code.
writer.using context manager is added to _get_subnets so that segment
plugin's _notify_subnet_updated handler that calls to _get_subnets
doesn't use the facade-less context.session that in specific cases may
cache old models from previous sessions when used in mixed
facade/facade-less environment.
This patch bumps SQLAlchemy minimal requirement to >= 1.2.0 because
pending_to_persistent event didn't exist before this version. It could be >=
1.1.0 if not for the fact that all 1.1.x releases have a bug that results in
breakage of test_update_with_none_and_own_mac_for_duplicate_ip due to obscure
import ordering issue in the library.
(The issue is fixed by https://github.com/zzzeek/sqlalchemy/commit/
63ff0140705207198545e3a0d7868a5ba8486e93)
Partially-Implements: blueprint enginefacade-switch
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Co-Authored-By: Michael Bayer <mike_mp@zzzcomputing.com>
Depends-On: If4b28110f460f6ac77ace1bbb02967ea986d4cab
Depends-On: I9f1e76cb24838533572b5fbe269ff96a24ce4af1
Change-Id: I0d65d19204da8ce30addfa5faff68544534b7853
Listen for foreign key changes and expire related relationships.
With this, we can remove OVO code that refreshes / detaches models on
each fetch. The patch also removes a bunch of expunge calls in plugin
code.
writer.using context manager is added to _get_subnets so that segment
plugin's _notify_subnet_updated handler that calls to _get_subnets
doesn't use the facade-less context.session that in specific cases may
cache old models from previous sessions when used in mixed
facade/facade-less environment.
This patch bumps SQLAlchemy minimal requirement to >= 1.2.0 because
pending_to_persistent event didn't exist before this version. It could be >=
1.1.0 if not for the fact that all 1.1.x releases have a bug that results in
breakage of test_update_with_none_and_own_mac_for_duplicate_ip due to obscure
import ordering issue in the library.
(The issue is fixed by https://github.com/zzzeek/sqlalchemy/commit/
63ff0140705207198545e3a0d7868a5ba8486e93)
Partially-Implements: blueprint enginefacade-switch
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Co-Authored-By: Michael Bayer <mike_mp@zzzcomputing.com>
Change-Id: I18c6794f99d2847c208dfd6e9eb187d53b657a05
This patch is a fix for the issue B101 assert_used found by bandit
scanner:
https://docs.openstack.org/bandit/latest/plugins/b101_assert_used.html
As assert functionality could be turned off by the -O option for python
interpreter, assert statements were replaced to raise AssertionError.
The main reasoning for that is rest code after assert could be not
ready for values that assert filters.
Change-Id: I92a871b1b496c96c0c76cb37e4dda51bfc007e38
Related-Bug: #1759250
It's remnant of enginefacade migration that seems to be unused in the
tree and deprecated since Ocata.
Depends-On: Idb08f9bd1b0b9a97915fb9099c2c66ae0d847a94
Depends-On: I676c421adc0b3bd8c47629cf8768cf86fd92b7ab
Depends-On: I8b14dbcea341755a5c2b02b35b809d6b364de4c7
Change-Id: I71c3f09475675bbbd28c72cea50b5c1c67b8ae30
neutron-lib contains the OVO exception module. This patch switches all
such reference's over to lib's version of the module and removes it
from neutron.
Change-Id: Ic3d435eef51f970598bf50486894838498b2d438
Note: We don't currently have any out-of-tree consumers of this module.
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
The reraise_as_retryrequest decorator is already defined in
neutron_lib.db.utils and is no longer used in neutron [1].
This patch removes the dead code from neutron.
[1] http://codesearch.openstack.org/?q=reraise_as_retryrequest
Change-Id: Ie453dea132bc8b7f195b68c22a7ec150765102b2
OVO detaches DB objects from the session and then holds onto
them for the life time of the OVO object. Simultaneously we attach
a weakref to the object to the session to load relationships for it
on commit.
If GC hasn't eliminated this object by the time we try to load
relationships, we end up trying to load rels for a detached object,
which implicitly tries to add it back into the session. This fails
if there is another copy in the session.
This patch just skips loading detached objects because we don't need
relationships loaded for them anyway.
Closes-Bug: #1683469
Change-Id: Ida3272562edb6608c6bfd7e11267159d6565a0c7
During commit processing, sqla automatically flushes objects.
However, it does so after dispatching the before_commit event,
while Neutron expects it happens before the event. This
counter-intuitive behaviour might or might not be considered
a bug in sqla. Anyway, it's better to be safe in our side.
Closes-Bug: #1678726
Change-Id: Iceaf178be2885b0299d6d281983e9f7f8ae6def0
Handle interim flushes by tracking 'new' objects in session.info
until commit time. This is necessary because new objects will no
longer be in 'session.new' during the before_commit event if flushes
have occurred.
Don't load relationships until final commit in nested commits. This
ensures we are on the outermost commit that will end the session
before loading up all of the relationships.
Partially-Implements: blueprint enginefacade-switch
Change-Id: Id0f79ebaafc446bb28363d281249f02eacd1e28d
This event handler will ensure that all eagerly loaded
relationships are loaded on a newly created sqla object.
This will allow us to continue our pattern of calling
make_whatever_dict outside of the transaction context
manager after the switch to the new engine facade
managers.
Co-Authored-By: Mike Bayer <mike_mp@zzzcomputing.com>
Partially-Implements: blueprint enginefacade-switch
Change-Id: I144fed595a740c29fe58ec75fe8dd0a1f1ca4523
Yes, they are innocent; yes, we log them with DEBUG level. But they
scare people, they make people judge the software, they are bad PR, and
they are probably redundant, string representation of the exception
should be enough. If there is an issue that will result in retry
attempts used without remedy, only then we should spill tracebacks and
scare users.
Guidelines: https://specs.openstack.org/openstack/openstack-specs/specs/log-guidelines.html#stacktraces-in-logs
Partial-Bug: #1673249
Change-Id: I791a52161635da64b1840c01ee6cf56a51bc7400
Add support for oslo.db's DBError to is_nested_instance
so we can check for IntegrityErrors in DBErrors easily
that come back from oslo.db.
This will enable the patch that depends on this to detect
port creation/network delete races very easily and retry
them.
Change-Id: I617f2549caced6547d478caba968710ad8f913b3
Register all sqlalchemy events through a new function in
neutron.db.api so we can keep track of active events and
ensure all are removed at the end of each test run.
Without this, an instance of a plugin may be left around
with the only reference to it existing in SQLAlchemy, where
it will receive events for tests unrelated to it and potentially
interfere.
Change-Id: I8e93eb4e8ef5a13f015db9cd20e44941cdcb72ef
Introduce get_reader_session() and get_writer_session()
and replace get_session() with them.
Mark get_session as depricated.
Stop using get_engine from legacy facade. Use writer engine for
places where it is required.
Partially-Implements blueprint: enginefacade-switch
Change-Id: I28b741bfa27bf04cbe273586e6e3e00e14fbe683
OVO object create() raises exception NeutronDbObjectDuplicateEntry
on seeing duplicate entries which should be handled by retrying the
transaction.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Closes-Bug: #1627811
Change-Id: I2b311326c2caa7500aa9318d0b5bf753cf0eb543
Options are for operators, not for the code. We should switch to public
functions provided by the library, to get what we want.
Change-Id: Id124dc6743d83c62d264ecc6b01a6b4006e44f35
The patch makes the module to not assume the options are registered by
some other code, that may never be executed, f.e. when it's part of a
single test case execution.
Change-Id: I7eefb32e6f34d732bda681919b434a96b8ad36da
Closes-Bug: #1640782
Change I6f0d108fc00cc7fcfaac1bd8f1b0468fcfbaa0dd deprecated
dispose_pool(). Instead of this context_manager.dispose_pool() can be
used.
Change-Id: I532e37b42f399ab6ca536f360772b1ccbcfb818a
This function was deprecated in change
Id38c06aee428200a061c59a984b59b81b24056e3
get_engine() can be called directly with
context_manager.get_legacy_facade().get_engine()
Change-Id: I7ef45c7ced65a3a3746e616c7b796a34b3551d3c
Related-bug: #1520719
Marked as deprecated in Change I151ffcf47926f5ac66e452974f87e8bc2a906151
Use is_retriable instead of is_deadlock.
Change-Id: I895d288c2fcbb225ac2c116f378fbbb43b052549
Notifier initialization moved to osprofiler. Added call
to init_from_conf method that does it.
Fixed osprofiler admin context.
Change-Id: Ic5c7bf72cafe077f6a77639fa439a61d39f89126
Depends-On: I77d0d3b5e9e725507916724dcd28d5fbc6f7f5c7
This decorates the quota system operations with
the retry decorators. This will help significantly
with the bug this marks as closed since operations
in the quota engine after commit should no longer
trigger retries of the full API operation.
The logic to find the args in the decorator had
to be adjusted to deal with functions already decorated.
This just uses the getargspec function from pecan that
deals with decorated functions.
Partial-Bug: #1612798
Closes-Bug: #1596075
Change-Id: Ib786117dcea08af75551770ea4c30d460382b829
Retrying mutating operations at the API layer caused a
couple of problems. First, when components would call
the core plugin using the neutron manager, they would
have to handle the retriable failures on their own or
undo any work they had done so far and allow retriable
failures to be propagated up to the API. Second, retrying
at the API makes composite operations (e.g. auto allocate,
add_router_interface, etc) painful because they have to
consider if exceptions are retriable before raising
fatal exceptions on failures of core plugin calls.
This patch begins the process of moving them down to the
core operations with a new decorator called
'retry_if_session_inactive', which ensures that the
retry logic isn't triggered if there is an ongoing transaction
since retrying inside of a transaction is normally ineffective.
Follow-up patches apply them to various parts of the code-base.
Additionally, the args and kwargs of the method are automatically
deep copied in retries to ensure that any mangling the methods
do to their arguments don't impact their retriability.
Finally, since we are leaving the API decorators in place for now,
the retry logic will not be triggered by another decorator if an
exception has already been retried. This prevents an exponential
explosion of retries on nested retry decorators.
The ultimate goal will be to get rid of the API decorators entirely
so retries are up to each individual plugin.
Partial-Bug: #1596075
Partial-Bug: #1612798
Change-Id: I7b8a4a105aabfa1b5f5dd7a638099007b0933e66
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.
Fix miscellaneous other deprecations.
Uses constants instead of l3_constants when importing
neutron-lib constants.
Co-Authored By: Henry Gessau <gessau@gmail.com>
Co-Authored By: Gary Kotton <gkotton@vmware.com>
Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
get_engine() can be called directly with
context_manager.get_legacy_facade().get_engine()
Related-bug: #1520719
Change-Id: Id38c06aee428200a061c59a984b59b81b24056e3
Mike Bayer