There already exists a unique constraint on the same columns, making
an additional index redundant.
Closes-Bug: #1988421
Change-Id: I8236ec1f685a3ae7c503d3ff8148138a875d702a
This reverts commit 6fa3d8019f.
Reason for revert: https://bugs.launchpad.net/neutron/+bug/2025126
A new SLQAlchemy/alembic (master branch) version reverted the
behaviour fixed in the patch that is being reverted.
Closes-Bug: #2025126
Change-Id: I9322589d973d9469d92878a505de6133978ade84
This patch changes all RBAC relationship method to "joined". This change
enforces that the RBAC associated registers are loaded along with the
parent resource. The rationale of this change is to be able to control
the SQL query executed; the subquery cannot be directly managed by
Neutron.
It is very usual to create the RBAC rules from one single project that
is usually the adminitrator project. That means all RBAC rules will
belong to it. Before this change, the SQL subquery performed to
retrieve the RBAC entries was this (from a network query):
SELECT networks.id AS networks_id
FROM networks LEFT OUTER JOIN networkrbacs ON networks.id =
networkrbacs.object_id
WHERE networks.project_id = 'bd133e2c499c4bf8aeb16206e31c3c20'
OR networkrbacs.action = 'access_as_external'
AND networkrbacs.target_project = 'bd133e2c499c4bf8aeb16206e31c3c20'
OR networkrbacs.target_project = '*'
OR networks.project_id = 'bd133e2c499c4bf8aeb16206e31c3c20'
OR networkrbacs.action IN ('access_as_shared', 'access_as_readonly')
AND (networkrbacs.target_project = 'bd133e2c499c4bf8aeb16206e31c3c20'
OR networkrbacs.target_project = '*');
This SQL result has a very low cardinality; that means there are many
duplicated registers. For example, with 10 external network, 1000
projects and 2500 RBAC rules, this query returns 1.4 million rows.
Instead if a "GROUP BY resource_id" (in this case network_id) clause is
added, the number of rows is reduced to 10 (considering this project
has a RBAC per network).
In order to introduce this "GROUP BY" clause, this patch is changing
the loading method. The clause is added in a neutron-lib patch [1].
This change by itself does not improve the query performance. The
neutron-lib patch is needed too. Although this patch does not modify
que SQL query results, the tests added will prove that the neutron-lib
patch does not introduce any regression.
[1]https://review.opendev.org/c/openstack/neutron-lib/+/884878
Closes-Bug: #1918145
Change-Id: Ic6001bd5a57493b8befdf81a41eb0bd1c8022df3
The alembic comparator needs the same class to compare the
database model with the loaded SQL schema.
Closes-Bug: #2020050
Change-Id: I4feab3e55f2e38a9eca70d4ca755b4c2974346d5
in [1] a lock was introduced with the goal of preventing subnets from
being deleted while ports are being created in them in parallel.
This was acheived by aquiring an exclusive lock on the row of the
subnet in the Subnet table when adding/modifying a port or deleting
the subnet.
However as this was a exclusive lock it also prevented concurrent port
modifications on the same subnet from happening. This can cause
performance issues on environment with large shared subnets (e.g. a
large external subnet).
To reduce the lock contention for this case we split the lock in two
parts:
* For normal port operations we will aquire a shared lock on the
row of the subnet. This allows multiple such operations to happen in
parallel.
* For deleting a subnet we will aquire an exclusive lock on the row of
the subnet. This lock can not be aquired when there is any shared
lock currently on the row.
With this we maintain the same locking level as before, but reduce the
amount of lock contention happening and thereby improve throughput.
The performance improvement can be measured using rally test [2].
(improving from 21 to 18 seconds).
Alternatively it can be tested using 250 parallel curl calls to create a
port in the same network. This improves from 113s to 42s.
[1]: https://review.opendev.org/c/openstack/neutron/+/713045
[2]: https://github.com/openstack/rally-openstack/blob/master/samples/tasks/scenarios/neutron/create-and-delete-ports.json
Closes-Bug: #2009055
Change-Id: I31b1a9c2f986f59fee0da265acebbd88d2f8e4f8
The method "get_network_address_scope" filters
"Subnetpool" with "address_scope_id" using an
exact match. Making the column indexed will improve
the query performance.
Closes-Bug: #1973726
Change-Id: Ib3f8e18ba28b277d5fa02dd386ca80a5a113c247
The method ``_port_filter_hook``, that is added in any "Port" SELECT
command, filters the database "Port" registers by "network_id", using
an exact match. This query speed will improve if this column is
indexed in the database engine.
Closes-Bug: #1973349
Change-Id: Ia20f96dc78ea04bb0ab4665e6d47a6365789d2c9
The view only relatinships must be specified in the DB model.
The following are fixed:
- QosNetworkPolicyBinding.port: "port" DB object will have a view only
copy of the "network" QoS policy.
- Subnet.rbac_entries: "subnet" DB object will have a view only copy
of the "network" RBAC policies.
Change-Id: I2396c0a23b00439d46faf4b29a082b3819528d89
Closes-Bug: #1927763
The field "in_use" is added to "subnet" DB definition. This DB
register column is a flag used to mark a register as in use
by other transaction. When a write DB transaction writes any
value on this field, the register is locked for any other
concurrent transaction. If two DB transactions try to set this
column at the same time, one of them will fail.
This DB lock is implemented in "subnet" and is used during the
subnet deletion and the port IP assignation, where all the port
network subnets are retrieved to provide an IP address on the subnet
CIDR.
As reported in the related bug, it was possible to assign an IP
to a port and, before the port creation command finished, delete the
subnet where the IP belonged. This patch introduces this subnet lock
during the IP assignation and at the beginning of the subnet deletion
process. At the end of both transactions, the DB engine checks if the
lock operation (write "in_use" column) is possible or the subnet
register was already requested by other DB transaction.
Change-Id: I45a724917389814e83400f5854ada175dfce2b7b
Closes-Bug: #1865891
Add a new DB table "network_subnet_lock". The primary key will be the
network_id. When a subnet is created, inside the write context during
the "subnet" object creation, a register in the mentioned table is
created or updated. This will enforce the serialization of the "subnet"
registers belonging to the same network, due to the write lock in the
DB.
This will solve the problem of attending several "subnet" creation
requests, described in the related bug. If several subnets with the
same CIDR are processed in parallel, the implemented logic won't reject
them because any of them will not contain the information of each other.
This DB lock will also work in case of distributed servers because the
lock is not enforced in the server logic but in the DB backend.
Change-Id: Iecbb096e0b7e080a3e0299ea340f8b03e87ddfd2
Closes-Bug: #1852777
This patch sets the MTU attribute to non-nullable, the code
that get the MTU and update the network in some methods can
be removed. If the MTU is empty before the pike version, it
is set to the default value of 1500.
Change-Id: Id4d738dde7fa4b7caccabad0aac542b82b4d7af1
Closes-Bug: #1842261
The "subnet" OVO does not allow to have an empty (None) "network_id" but
the DB "subnet" table allows to have this parameter empty (NULL) in the
database. In order to avoid any problem like the one described in the
bug, this patch ensures the database "subnet" register does always have
a "network_id" value and if a "network" register is being deleted, all
related "subnet" registers are checked first.
Change-Id: Iad210f0585b4201fdb87187b44a9b42267b58db4
Closes-Bug: #1839658
Basically, in case of concurrent requests creating ports
on *one* network, only one request succeeds, other requests
mostly getting 'standardattribute' update error as network
revision_number changed about 8 times for all port updates.
This patch increases network revision_number in database,
only when a port is created in a network, instead of each
update on port.
Change-Id: Idffb4edda616677b9b071644d3835c85052091a5
Closes-Bug: #1782421
This patch adds ``net-mtu-writable`` API extension that allows to write
to network ``mtu`` attribute.
The patch also adds support for the extension to ml2, as well as covers
the feature with unit and tempest tests. Agent side implementation of
the feature is moved into a separate patch to ease review.
DocImpact: neutron controller now supports ``net-mtu-writable`` API
extension.
APIImpact: new ``net-mtu-writable`` API extension was added.
Related-Bug: #1671634
Change-Id: Ib232796562edd8fa69ec06b0cc5cb752c1467add
Increment the revision number when RBAC policies are
changed since it impacts the calculation of the 'shared'
field.
Closes-Bug: #1708079
Change-Id: I4c7eeff8745eff3761d54ef6d3665cf3dc6e6222
Tag mechanism supports network, subnet, port, subnetpool
router resources only. This patch allow tag mechanism to support
resources with standard attribute.
Two old extenions are kept because of backward compatibility.
They will be removed in Queens release.
APIImpact: Tag is supported by resources with standard attribute
DocImpact: allow users to set tags on resources with standard attribute
Change-Id: Id7bb13b5beb58c313eea94ca03835d3daf5c94bc
Closes-Bug: #1682775
neutron-lib now contains the API definitions for neutron's core
resources. This patch removes the constant core resource and collection
variables and uses them from lib. Subsequent patches will consume the
actual core resource attribute definitions.
NeutronLibImpact
Change-Id: Ia9afdf620cd538b2aa420593277d6403a45c996b
All of the additional lookup logic was essentially duplicating
a relationship property of 'load_on_pending=True', which tells
SQLAlchemy to lookup the relationship during object creation [1].
So we can dump all of this logic and just use that option.
1. http://docs.sqlalchemy.org/en/latest/orm/relationship_api.html
#sqlalchemy.orm.relationship.params.load_on_pending
Related: blueprint push-notifications
Change-Id: I0e495a50f5cab9b6449825039d7683d77de1e763
The Subnet model has a relationship to Network that is only
used for bumping the revision of the network. However, loading
the network loads all of its relationships as well so we end
up wasting a bunch of time loading segments, DHCP agents, etc all
for no reason. During a subnet update, we get the subnet from the
DB several times (policy engine, before update, after update,
and dhcp agent scheduler) so the slowness in get_subnet is amplified.
The recent switch to subqueries[1] has made loading all of this
unrelated data much more expensive since each relationship requires
a query, which was likely what brought this problem to the surface.
This adjusts the relationship to just load the standard attr object
of the network directly so we can still bump the revision of the network
without having to load the network itself.
This resulted in a ~20%-30% reduction of time spent in single update-subnet
operations on my dev environment (1sec to 650-850ms).
1. I6952c48236153a8e2f2f155375b70573ddc2cf0f
Closes-Bug: #1665967
Change-Id: Ie5f956ddf0704a5dde6e4f801a446f48fea5f697
This relationship doesn't serve a purpose and it results
in SQLAlchemy trying to set the network_id column on ports
to NULL when attempting to delete a network with ports in it.
This results in the following exception:
IntegrityError: (1048, u"Column 'network_id' cannot be null")
This exception is not treated as a DBReferenceError by olso.db
which makes detecting races between ports being created on a
network being deleted very noisy in the logs since oslo.db treats
this as a generic DBError.
By removing the relationship, deleting a network will now just
result in a boring standard foreign key constraint failure which
will propagate from oslo.db as a DBReferenceError that is easy
to catch.
The patch that depends on this one adjusts the delete network logic
to capture these ReferenceErrors.
Change-Id: I4166a9bbaded57a77576400929a150123b8f1b2e
This switches to the use of subqueries for 1-m relationships
which will result in a higher constant query factor but will
eliminate the potential for cross-product explosions.
Closes-Bug: #1649317
Change-Id: I6952c48236153a8e2f2f155375b70573ddc2cf0f
Remove The following _MAX_LEN constants from
neutron/api/v2/attributes.py and use the corresponding DB field size
constants from neutron_lib.db.constants instead.
NAME_MAX_LEN --> NAME_FIELD_SIZE
TENANT_ID_MAX_LEN --> PROJECT_ID_FIELD_SIZE
DESCRIPTION_MAX_LEN --> DESCRIPTION_FIELD_SIZE
LONG_DESCRIPTION_MAX_LEN --> LONG_DESCRIPTION_FIELD_SIZE
DEVICE_ID_MAX_LEN --> DEVICE_ID_FIELD_SIZE
DEVICE_OWNER_MAX_LEN --> DEVICE_NAME_FIELD_SIZE
In alembic migration scripts, the raw numerical value is used.
For more information, see:
http://lists.openstack.org/pipermail/openstack-dev/2016-October/105789.html
NeutronLibImpact
Change-Id: I734890372584fe27e5d6ec38c0cad2de882ff11c
Loading subnets as part of the networks list and networks
as part of the subnets list appears to have a significant
impact when the network has tags and the subnets have
extra routes entries. This is even further compounded by
the network having rbac entries (likely due to the subnet
inheriting the RBAC entries of the network with the custom
join condition in the model).
This patch converts rbac_entries on both subnet and network
to use a subquery and converts the network and subnets
relationships on the subnet and network models (respectively)
to use subqueries as well.
On my dev environment after running the script in the report,
a network list took 5 minutes. Converting just the rbac_entries
or just the network/subnet relationship to subqueries reduced it
to 3-5 seconds. Converting both (as this patch does), reduces it
back down to a couple of hundred milliseconds (normal perf of my
development env with the current network count).
Subqueries will just cost us a constant number of queries and
won't scale up with result count so this should not impact scalability
in any way.
None of these fields are queryable from the API, so we don't need
to worry about breaking queries against the models.
Partial-Bug: #1649317
Change-Id: Ic1947e3d78d58a79b21344b10cb7ab0e573e419f
This adds an order_by clause to the fixed_ips relationship
on the port object to ensure that the fixed_ip ordering is
consistent between a create, an update, and a get request
for a port. Without it we were at the mercy of the sql backend
to determine how it felt like ordering them on the join condition.
Closes-Bug: #1630410
Change-Id: I523e0ab6e376f5ff6205b1cc1748aa6d546919cb
Those objects are intentionally not integrated into the database code so
far. This is to quicken access to their definitions to implement
push-notifications for security groups and ports.
The object embeds segmentation information in addition to what's
available through the model. Specifically, binding_levels field exposes
all ml2 binding levels, that from their side load corresponding network
segment object. The order for level objects in binding_levels list field
is guaranteed to be in the order of level. So the consumers can eg.
access the bottom binding info with:
port_obj.binding_levels[-1].segment
For PortBindingLevel object, we want to expose segmentation info. This
is achieved through a 'segment' ObjectField. The database model itself
contains segment_id too. There is no reason though to expose it for
Level object in two places (one as a model field, another one through
the ObjectField), so we avoid adding ID field. The base class that
handles loading for ObjectField based synthetic fields was assuming that
objects always have a field per model attribute, so it needed a slight
adjustment to support this case, where we extract foreign_keys
attributes from the model itself if the field is not present on the
object.
Partially-Implements: blueprint adopt-oslo-versioned-objects-for-db
Partially-Implements: blueprint push-notifications
Change-Id: I25de14e42e345d9235dbf4097c298ef5d606de51
Co-Authored-By: Martin Hickey <martin.hickey@ie.ibm.com>
Co-Authored-By: Rossella Sblendido <rsblendido@suse.com>
Co-Authored-By: Manjeet Singh Bhatia <manjeet.s.bhatia@intel.com>
Co-Authored-By: Brandon Logan <brandon.logan@rackspace.com>
Co-Authored-By: Victor Morales <victor.morales@intel.com>
This adds a way for standard attribute models to declare
the API resources they show up in. It then adds a utility
function to the standard_attr module to grab a map of all
API resources and their corresponding models.
This can be used by any processing code that wants to add
fields to standard attribute resources.
This also adjusts the existing extensions to leverage this
new functionality.
Partially-Implements: blueprint add-neutron-extension-resource-timestamp
Change-Id: Idc8923d0e983fcb0690f8cb5b55a5aff8690154f
This is the Neutron side of the unaddressed ports blueprint. In order
to allow unaddressed ports, Nova wants the port to explicitly say it
is okay that it doesn't have any IP addresses.
In Neutron, an unaddressed port is one that was created by explicitly
passing [] in fixed_ips to create the port. A new DB field is added
to the port to distinguish the unaddressed port case from the deferred
IP allocation case where routed networks is involved.
Change-Id: Ia61af4c14e955697a7d3fcc0bf4826a6d9475c98
Implements: blueprint vm-without-l3-address
APIImpact: port now has ip_allocation attribute, set on port create
The model_base file is going to move to Neutron lib in
I2087c6f5f66154cdaa4d8fa3d86f5e493f1d24d9. This will mainly leave
behind only the standard attributes related stuff so the name
'model_base' won't make much sense. This moves the standard attribute
related things into its own module so model_base can conceivably be
eliminated entirely.
Change-Id: Icaf3220fbc5723f2b5421a494371ef274d7073c7
Remove deprecation warnings for various constants
and exceptions that have moved to neutron_lib.
Fix miscellaneous other deprecations.
Uses constants instead of l3_constants when importing
neutron-lib constants.
Co-Authored By: Henry Gessau <gessau@gmail.com>
Co-Authored By: Gary Kotton <gkotton@vmware.com>
Change-Id: Ib0e8ff5c3e23677c1009241a1818cbc8a3430c38
This patch uses revision_number in database. When creating
a subnet in a network, the revision_number of the network
will be increased. That will prevent overlapping CIDR
(overlapping CIDR means some subnets' cidrs are overlapping)
on *one* network.
Basically, in case of concurrent requests creating subnets
on *one* network, only one request successes, other requests
needs retrying request.
Change-Id: Id6548535075bed87a4b36e1462db546ab9163f29
Closes-Bug: #1532695
This patch addresses all the DeprecationWarning errors related
to constants in neutron.db.models_v2.
Change-Id: I54402d4441b3ccb15bab63adf830fa196bba62c3
This adds the logic to increment the revision numbers
for objects whenever there are changes and it exposes
the revision number via a field in the API.
This is handled with a new default service plugin that
subscribes to DB events and bumps revision numbers for
any objects that were modified.
It also handles the logic for bumping the revision number
of a parent in a relationship where the children aren't
top-level neutron objects that would be tracked individually.
This is accomplished with a 'revises_on_change' attribute
on the child models that the service plugin will use to
find the parent and bump its revision.
API tests are included to test the revision numbers
added to each standard attribute enabled object.
Partially-Implements: bp/push-notifications
Change-Id: I476d3e03c8ee763cc4be6d679fe9f501eb3a19b5
This patch set aims to move all the code related to DNS integration
from the DB core plugin to the DNS ML2 extension module.
By doing this, this patchset removes the dns related code in
db_base_plugin_v2 and the dns exteions module talks with core plugin
only through the method extension_manager and apply_dict_extend_functions
By properly implementing the generation of the dns_assignment attribute
for ports in the DNS ML2 extension module, this patchset also fixes
https://bugs.launchpad.net/neutron/+bug/1579977
Change-Id: I63afb1a1bfeeb14eefb54681dc64959144deeb25
Closes-Bug: #1579601
Closes-Bug: #1579977
Today, existing networks may not reflect MTU configured for
neutron-server, if they were created when neutron-server was using
different MTU setup for its infrastructure, or when it was using bad
default values for network MTUs (specifically, before Mitaka, all networks
were getting MTU = 0 by default, disabling both advertisement and data
path MTU size enforcement).
This patch stops persisting MTU in the database on network create and
instead calculate it on every network resource fetch.
DocImpact Now changes to MTU configuration options immediately affect
existing network MTUs, not just new networks.
UpgradeImpact Existing networks with invalid MTU persisted in database
may change their MTU values to reflect configuration.
Change-Id: Iee4f5037bf10b73ba98464143b183aacb59c22f2
Closes-Bug: #1556182
There is a bug in pep8, when 'select' used, it omits all default checks
and runs only those specified by 'select'. We got hit by this issue
since I2d26534230ffe5d01aa0aab6ec902f81cfba774d was merged which lead to
almost no static checks in pep8 job.
Also note that off_by_default decorator has no effect for now because
factory in hacking is triggered after ignored checks are collected.
There will be a follow-up patch for that in order to make pep8 doing
its job quickly.
[1] https://github.com/PyCQA/pycodestyle/issues/390
Related-Bug: 1594756
Change-Id: I8e27f40908e1bb4307cc7c893169a9d99f3433c4
Zuul
yatinkarel