Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
some remaining ones in miscellaneous directories.
Also cleanup any remaining code that I missed in this
series, or has changed since I started.
Trivialfix
Change-Id: I17b4779020a7bfb369c3e721ab6638cd4a6ab50c
This patch removes the ``IpAddressAllocationNotFound`` exception. This
exception was raised when a IPAM register was called to be deleted
but not found.
As reported in the LP bug, this IPAM register deletion can be called
several times if a port fails during the creation. The IPAM register
deletion calls the DB deletion but doesn't raise any exception if the
register does not exist. The code ensures the IPAM register is
deleted and there is no need to fail if it is not present anymore.
This patch also removes the exception catch and try in "update_port",
that was added in [0] as a fix for [1]. That was added because the
subnet deletion code involved a port update call [2] during the
IP allocation deletion, if any port was still present in the subnet.
Since [3], this code is not needed because the subnet deletion does
not call a port update anymore.
[0]https://review.opendev.org/c/openstack/neutron/+/373536
[1]https://bugs.launchpad.net/neutron/+bug/1622616
[2]https://github.com/openstack/neutron/blob/pike-em/neutron/db/db_base_plugin_v2.py#L1017-L1018
[3]https://review.opendev.org/c/openstack/neutron/+/713045
Closes-Bug: #1965807
Related-Bug: #1954763
Related-Bug: #1622616
Change-Id: I5b96b3a91aacffe118ddbb91a75c4892818ba97a
As spotted in Focal testing patch [0], pep8 test fails with many
C0321 false-positives, reported in pylint as current version does not
support python 3.8 [1]
Use a newer version of pylint and astroid, fixing or disabling some of
the new checks: no-else-*, unnecessary-comprehension, import-outside-toplevel
[0] https://review.opendev.org/#/c/738163/
[1] https://github.com/PyCQA/pylint/issues/2737
Change-Id: Ie646b7093aa8634fd950c136a0eba9adcf56591c
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
1.It is best not to use 'netaddr.IPSet.add',
because _compact_single_network in 'IPSet.add' is quite time consuming
2.When the current address pool does not have enough addresses,
all addresses are allocated from the current pool,
and allocations are continued from the next address pool
until all addresses are assigned.
Change-Id: I804a95fdaa3552c785e85ffab7b8ac849c634a87
Closes-Bug: #1813253
Create a method for bulk assignment of IP addresses within the ipam
driver, to support bulk creation of ports.
This also changes the logic for how the window of available IP addresses
to assign from is calculated within the neutrondb IPAM driver. The
Python random module is used to produce a statistically sampled set of
IP addresses out of the set of available IPs; this will facilitate
collission avoidance. When requesting multiple IP addresses the
selection window sized is increased significantly to ensure a larger
number of available IPs, but caps are placed on the amount to make sure
we do not transgress system limits when building pools of IPv6
addresses.
Change-Id: Iad8088eaa261b07153fa358ae34b9a2442bc2a3e
Implements: blueprint speed-up-neutron-bulk-creation
Try to give it a large random chance during generate IP
address.
The DB retry mechanism change moved to neutron-lib:
I5ad139bdfb3ae125658b36d05f85f139a1b47bee
Closes-Bug: #1777968
Change-Id: I2acc9c720b39271bde2a89da4a66958f7aba5b7d
Instead of using old _get_subnet() method new method
_get_subnet_object() will now be used in neutrondb_ipam
driver and because of that it will use Subnet OVO object.
Change-Id: Ie9876064da989842576eeb30804b6a991d4d2937
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
Usage reader and writer for db operations.
In Neutron code we have common situation like:
with context.session.begin():
context.session.add(obj)
self._make_obj_dict(obj)
With new enginefacade we change context.session.begin() for
db.context_manager.writer(reader).using(context).
When object leaves this with-block, its reference to session is
cleared because session is discarded. To use this object later to
load some data from its dependencies, we have to provide different
session to work in. To solve this obj either can be moved under
with-block or we have to do context.session.add(obj) one more time
to be able to load relations.
This change also switches to usage of new enginefacade for some db
operations with ports, in order to pass unit and functional tests.
Partially-Implements blueprint: enginefacade-switch
Change-Id: Ia15c63f94d2c67791da3b65546e59f6929c8c685
This patch introduces and integrates Oslo-Versioned Objects for
IpamAllocationPool,IpamAllocation and IpamSubnet model classes.
Change-Id: Ib7d86dd10e92b5e2baeef0e69620c8688859fe7d
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Co-Authored-By: Anindita Das <anindita.das@intel.com>
Co-Authored-By: Mohit Malik <mohit.malik@intel.com>
Co-Authored-By: Victor Morales <victor.morales@intel.com>
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
This code is not being used outside neutron tree so it's
safe to change the api to use context instead of session.
This change is required for implementing Ipam OVO and is
consistent with rest of the DB API code in neutron.
Change-Id: I600ce4384632bf013846e8aa1618acb126f7c598
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
If a subnet is removed after lookup but before IP allocation
insert, we can end up getting a DBReferenceError in the IPAM
system. Since this can occur on any flush we force a flush
with a session context manager to get the DBReference error
in a desired spot (in multi-writer galera this is a deadlock
much later, which we retry). We then convert it to SubnetNotFound,
which is an expected exception.
Closes-Bug: #1623402
Change-Id: I65632b174f8692cd465c721e285526827842a740
With the in-tree pluggable IPAM driver, IPAM rollback tries to use the
DB session after it has been rolled back due to an exception. This
driver doesn't need roll back, so fix this by adding a method to the
driver signalling that rollback shouldn't be attempted.
Change-Id: Ic254789e58a8a51cd1aa943cb71de12410f4c0a7
Closes-Bug: #1603162
Related-Bug: #1516156
Reference IPAM driver skipped subnet_update processing
if allocation_pools in subnet request are blank.
Built-in ipam implementation allows to update allocation pools
to blank value (i.e. clean up allocation pools for subnet).
To make Reference ipam driver consistent with built-in ipam
implementation allocation_pool check was removed.
Included UT to verify that allocation pools can be updated to blank
value now.
Change-Id: I0654e46d4bc60f6cf51ffddeead238dd4f064db2
Closes-Bug: #1605277
This patch removes unreachable code that rebuilds IP address
availability data and persists in the database. This is
ultimately derived data that is now computed in memory and never
persisted. The code being removed is dead code and does not
include the contract migration for removal of the
IPAvailabilityRange models.
Change-Id: I96cb67396b8e0ebbe7f98353fad1607405944e44
Partial-Bug: #1543094
Preserve sequential IP allocation for DHCP ports to
avoid interferring with templating systems like heat
that expect a freshly created subnet to have higher
numbered IP addresses available for resources.
The change to randomized allocations was breaking
heat templates that tried to use a specific address
in the subnet that didn't use to be near where DHCP
ports were allocated (e.g. the 10th address).
Change-Id: I4dbda44460adc873b2e4dc1638a34bfac9bb1fc4
This patch computes IP availability in memory without locking on
IPAvailabilityRanges. IP availability is generated in memory, and
to avoid contention an IP address is selected by randomly
selecting from within the first 10 available IP addresses on a
subnet. Raises IPAddressGenerationFailure if unable to allocate an
IP address from within the window.
Change-Id: I52e4485e832cbe6798de6b4afb6a7cfd88db11e2
Depends-On: I84195b0eb63b7ca6a4e00becbe09e579ff8b718e
Closes-Bug: #1543094
Using nested transaction in reference ipam driver causes rewriting
original exception due failure to restore to safepoint in case of
DeadLock.
DBDeadlock exception gets replaced with DBAPIError. It prevents db_retry
wrapper from working correctly.
This patch removed nested transaction block from reference ipam driver.
Change-Id: Ib710e87b0132aae3cd3afd12c5448961f1a3b25c
Partial-Bug: #1571666
Commit 6ed8f45fdf had some negative impact
on concurrent ip allocations. To make ipam driver aware of subnet
updates (mostly for thirdparty drivers) ipam driver is always called with
allocation pools even if pools are not changed.
Current way of handling that event is deleting old pools and creating
new pools. But on scale it may cause issues, because of this:
- deleting allocation pools removes availability ranges by foreign key;
- any ip allocation modifies availability range;
These events concurently modify availability range records causing
deadlocks.
This fix prevents deleting and recreating pools and availability ranges
in cases where allocation pools are not changed. So it eliminates
negative impact on concurency added by always calling ipam driver on
subnet update.
This fix aims to provide backportable solution to be used with
6ed8f45fdf.
Complete solution that eliminates concurrent modifications in
availability range table is expected to be devivered with
ticket #1543094, but it will not be backportable because of the scope of
the change.
Change-Id: I29e03a79c34b150a822697f7b556ed168a57c064
Related-Bug: #1534625
Closes-Bug: #1572474
This fixes the IP driver update_allocation_pools to use
the session on the subnet rather than generating a new one.
Generating a new one was causing the allocation pool
creation to be uncommitted since nothing was ever starting
a transaction on the session before it added allocation pool
objects to the session. This resulted in them never being
committed to the database so the subnet would become unusable
after an update.
This also adds a very simple unit test to ensure the
subnet still works after an update.
Closes-Bug: #1573443
Change-Id: I1c80a01f475760b9cd405fc89ee2df3a4a1c26c4
The IP allocation failure constructions weren't being constructed
before being passed to RetryRequest so they would fail to be reraised
after the retries were exhausted.
This patch just constructs them and adjusts a few of the unit tests
to execute the full retry life-cycle.
Closes-Bug: #1555387
Change-Id: I1c1ea488886ee3f4790eaa53911ea5845f327334
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
Existing locking mechanism 'select for update' causes
deadlocks with galera multi-writers.
Replaced locking rows with compare-and-swap approach.
Compare-and-swap verifies that row is not changed by
another thread before updating/deleting it.
Filter-and-update and filter-and-delete are used.
They return count of affected rows.
If count of affected row is less than expected,
then another thread already changed our row
and RetryRequest is raised.
Change-Id: I514cae0fa43033433ec2982bcf3726e02e6692bf
Closes-Bug: #1494351
This patch makes the necessary changes for IPv6 PD to function when
pluggable ipam is enabled with the reference driver.
Includes a unit test for this functionality.
Change-Id: I4227cc08fdd62922632629c424dbeb542a48a67f
Partially-Implements: blueprint ipv6-prefix-delegation
Introduces new Pluggable IPAM backend.
IP/subnet allocation calls are sent to IPAM driver.
Calls to IPAM Driver are considered as call to third-party
environment, so if any action fails, rollback action is called.
Removes associate_neutron_subnet step from interface and reference
driver. It is not needed any more because foreign key relationship
between IPAM subnet and neutron subnet was removed.
So IPAM subnet can store id of neutron subnet,
which is not created yet.
For now only reference IPAM driver is available.
Temporarily disabled test_ipam_pluggable_backend from
gate-neutron-python34. Current patch adds executing
parts of test_db_base_plugin_v2, which is not py34 compatible yet.
Might be enabled back once 204791 is merged.
Partially-Implements: blueprint neutron-ipam
Change-Id: Ic18461cf19d2eebf8fad5535eee0eb034959800e
An alternate pluggable IPAM implementation from the built-in one
in db_base_plugin_v2.
Modifies IPAM interface to allow passing context to driver and
introduces new interface method 'associate_neutron_subnet'.
Implements blueprint reference-ipam-driver
Change-Id: I2e1e9fc7994bf1157bcd34b7ea500eb30c61d9ab
Brian Haley