Commit Graph

781 Commits

Author SHA1 Message Date
Rodolfo Alonso Hernandez 6b6abb9698 Improve the SG RPC callback ``security_group_info_for_ports``
This method populates the SG rules in a dictionary. Each SG rule
inherits the "stateful" value of the SG. Prior to this patch, each
SG rule was isuing a database call to retrieve the SG register.

In this patch, the SG "stateful" retrieval is done in one database
query for all SG. That improves the performance of this method
reducing the database access to only one single call.

This improvement, as commented in the LP bug, affects to
ML2/LinuxBridge. ML2/OVS agent uses a cached RPC implementation
that not requires to perform any RPC call/database query.

Closes-Bug: #2045950
Change-Id: Iafd0419a1d1eeb25d5589edc2570ebf287450957
2023-08-21 15:46:13 +00:00
Rodolfo Alonso Hernandez 96223931ca Create a policy rule to control if a rule belongs to the default SG
The policy rule ``shared_security_group`` allows to create new policy
rules checking if a security group rule belongs or not to the project
default security group.

By default the behaviour has not changed. If an administrator wants
to prevent a non-privileged user from creating or deleting rules in the
default security group, the ``create_security_group_rule`` and
``delete_security_group_rule`` can be overriden. An example is provided
in the unit tests.

Closes-Bug: #2019960

Change-Id: I6c90b61df0e726ef07f177801069baf30c49de67
2023-10-09 14:11:55 +00:00
Zuul d32c5f8f32 Merge "Fix some new pylint "R" warnings" 2023-07-28 06:58:46 +00:00
Brian Haley 929b383743 Fix some new pylint "R" warnings
After updating pylint, it started emitting additional "R"
warnings in some cases, fix some of them.

  use-a-generator,
  unnecessary-lambda-assignment,
  consider-using-max-builtin,
  consider-using-generator,
  consider-using-in,
  use-list-literal,
  consider-using-from-import

Trivialfix

Change-Id: Ife6565cefcc30b4e8a0df9121c9454cf744225df
2023-07-18 18:06:51 -04:00
Zuul bce16f6f3d Merge "Delete sg rule which remote is the deleted sg" 2023-07-01 05:12:35 +00:00
elajkat 67a0b07287 Delete sg rule which remote is the deleted sg
Based on bug #2008712 if we have a security-group which
is the remote group of a 2nd security-group, the backend
never deletes the rule of the 2nd group which
remote_group_id is the original security-group.
By AFTER_DELETE event for each rule that has the
security_group_id as remote_group_id, we can make the
mech drivers do their work and delete these rules in the
backend.

One version of this fix was merged:
https://review.opendev.org/q/I207ecf7954b06507e03cb16b502ceb6e2807e0e7
and reverted due to #2019449:
https://review.opendev.org/q/I077fe87435f61bd29d5c1efc979c2adebca26181

This patch is based on
https://review.opendev.org/c/openstack/neutron/+/876716/1

Closes-Bug: #2008712
Related-Bug: #2019449
Change-Id: I9e8ddfa26c5402fefd573b0e2ea5f3a57983ca35
2023-06-26 07:37:04 +00:00
Guillaume Espanel ce12b6ac19 Do not query neutron-rpc for sg rules upon sg deletion
This makes _clear_child_sg_rules look into the local resource cache
for sg rules that have to be deleted instead of querying the
neutron-rpc server.

Deleting a security group results in each agent of the region running
a bulk_pull query for all the rules in the security group against the
neutron-rpc. This is incurs a load on neutron-rpc, rabbitmq and the db
proportional to the number of agents and the number of security group
rules and has a noticeable impact on larger infrastructures.

However, when the cache is properly populated, we already have the
rules we should remove from the cache and don't need to ask neutron
server for them.

Closes-Bug: #2022360
Change-Id: I53e11f558a6cd84a02041758badddfe87a10c95c
2023-06-05 14:53:18 +02:00
yatinkarel 3e1e2d63b3 Enable rpc notifications only when rpc_workers >= 1
rpc_workers can be set < 1 with 'ovn' backend when no
other agent is running apart from ovn agents to
consume these rpc notifications.

Add and apply disable_notifications decorator on
methods which do rpc cast calls to agents, the
decorator makes the caller method execute only
when rpc_workers >=1. This patch not changing
default behavior and utilizes the rpc_workers config option
to enable rpc notification on resources updates only when
rpc_workers >= 1.

Also set rpc_workers=0 in ovn jobs to cover this scenario.

Closes-Bug: #1889737
Closes-Bug: #1992352
Change-Id: I700fe2cd422bc1eb8b5144ec116e7f0a60238419
2023-01-09 13:33:37 +05:30
Brian Haley 291eabb0b6 Fix some pylint indentation warnings
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/api.

Trivialfix

Change-Id: I1258b04f64a18036407e1d9de9ddca7472af0d11
2022-11-02 10:11:46 -04:00
Sahid Orentino Ferdjaoui 7ceb935da8 dhcp/rpc: retrieve network details with segments
When segment plugin is enabled, we should return segments details as
they are part of network.

Partial-Bug: #1956435
Partial-Bug: #1764738
Signed-off-by: Sahid Orentino Ferdjaoui <sahid.ferdjaoui@industrialdiscipline.com>
Change-Id: I1dab155bc812f8764d22e78ebb7d80aaaad65515
2022-09-01 14:48:54 +02:00
Slawek Kaplonski 06ddcaf436 Port update will trigger less notifications to the DHCP agents
After port update, DHCP agent will be notified about changes only if
one of the port's attributes related somehow to the DHCP will change.
Such fields are:
* fixed_ips,
* MAC address,
* dns_domain,
* dns_name,
* dns_assignment,
* extra_dhcp_opts.

In other cases there is no reason to send notifications to the agent.
This will results with less notifications to the DHCP agent and less
possibilities to race condition between DHCP and L2 agents while
switching ports from the DOWN to ACTIVE status and sending notifications
to nova.

Closes-Bug: #1982367
Change-Id: If7990bdec435af76ad2e88fd4ea2bc24a255fd5a
2022-07-21 12:10:03 +02:00
Brian Haley 357ee21d12 Remove import of 'imp' module
Use importlib.util module instead of imp, since it's
being deprecated:

  DeprecationWarning: the imp module is deprecated in favour of
  importlib; see the module's documentation for alternative uses

Had to change test setup to call super() first to get around a
config issue I was seeing locally, causing the entire class of
tests to fail:

  oslo_config.cfg.NoSuchOptError: no such option api_extensions_path
  in group [DEFAULT]

Closes-bug: #1981077

Change-Id: Ic171028a661c3f9f83f6758a57aaeab4450aa907
2022-07-11 14:19:54 -04:00
Zuul 3690795ab6 Merge "FUP: remove convert_to_sanitized_binding_profile_allocation" 2022-05-17 13:04:53 +00:00
Slawek Kaplonski cd8bf18150 [L3HA] Don't update HA router's ports if router isn't active on agents
In case when HA router isn't active on any L3 agent,
_ensure_host_set_on_port method shouldn't try to update port's host to
the host from which there was an rpc message sent, as this can be host
on which router is in the "standby" mode.
This method should only update port's host to the router's "active_host"
if there is such active_host found already.

Depends-On: https://review.opendev.org/c/openstack/requirements/+/841489

Closes-Bug: #1973162
Closes-Bug: #1942190
Change-Id: Ib3945d294601b35f9b268c25841cd284b52c4ca3
2022-05-13 15:05:47 +00:00
elajkat 701900ac33 FUP: remove convert_to_sanitized_binding_profile_allocation
convert_to_sanitized_binding_profile_allocation was added to Neutron
temporarily before [1] was merged and released in neutron-lib.

[1]: https://review.opendev.org/c/openstack/neutron-lib/+/813650
Related-Bug: #1922237

Change-Id: I953b96d97076cd6a80fff6e97e2fd956da737d46
2022-05-09 15:18:31 +02:00
Slawek Kaplonski 51d1899bac Handle properly InvalidScope exceptions to not return error 500
When new default policy rules and scope enforcement are enabled, Neutron
needs to handle properly not only PolicyNotAuthorized exception from
oslo_policy module but also InvalidScope exception.
This patch adds handling of that exception to the neutron policy
modules.

In the check() method from the neutron.policy module we are calling
ENFORCER.enforce() method with do_raise=False which means that
PolicyNotAuthorized isn't rasised. Unfortunately it seems that there is
bug in oslo.policy module and InvalidScope is raised even with
do_raise=False.
For now, lets workaround it in Neutron by properly handling InvalidScope
exception in the check() method.
This workaround can be cleaned when bug [1] will be fixed in
oslo.policy.

[1] https://bugs.launchpad.net/oslo.policy/+bug/1965315

Partial-Bug: #1959333
Change-Id: I973f8896248c8222031c53343bb53ce48254da74
2022-03-30 14:52:37 +00:00
Yang JianFeng 9b27020a65 [Agent Side] L3 router support ndp proxy
The agent side codes need consider three scenarios:
1. Non-dvr router. The all related rules are applied in
   qrouter-namespace
2. Dvr router with the local agent mode is dvr_no_external.
   The all related rules are applied in snat-namespace.
3. Dvr router with the local agent mode is dvr. In this scenario,
   The all related rules are applied in fip-namespace.

Change-Id: Ie8729586d318be4a673858021a0116e09e193522
Partial-Bug: #1877301
2022-02-25 12:42:13 +08:00
Slawek Kaplonski 8fd88fd223 Pass host parameter to the get_network_info method
Host parameter is needed there to filter subnets per segment when
segments plugin is enabled.
When dhcp agent requests informations about networks, and segments
plugin is enabled, subnets which belongs to the network are filtered out
based on the host passed as argument to the get_network_info() method.
But we never passed host to that method, even when we should e.g.
during the full sync of the DHCP agent, when it requests details about
each network.
This patch fixes that issue by passing host parameter to that method.

Closes-Bug: #1958955
Change-Id: Ib5eef501493f6735a47ea085196242a5807c4565
2022-02-01 08:28:15 +01:00
Slawek Kaplonski 1c1814aa6b Fix reference before assignment error in the dhcp_rpc module
In patch [1] method get_network_info was refactored and that causes
NameError in the DHCP agent when there is "network object passed in
kwargs and there are subnets with segments in network. See related bug
for details.

[1] https://review.opendev.org/c/openstack/neutron/+/820190

Closes-Bug: #1958955
Change-Id: Iad8d85c79f8b11a24b1bb1ca44c776e909b610c3
2022-01-26 12:50:19 +01:00
Rodolfo Alonso Hernandez 5710d3407b Improve DHCP RPC handler
Remove unnecessary DB retrieval operations from
"get_active_networks_info" method.

Partial-Bug: #1950662
Change-Id: I4ea7b86e3f544d5dddcdac562208bb8afd1fc36a
2022-01-15 07:58:25 +00:00
Zuul 5e5cc0e184 Merge "Improve DHCP RPC handler" 2021-12-07 00:23:26 +00:00
Zuul 9c74059c60 Merge "Add Local IP L2 extension skeleton" 2021-11-30 19:07:48 +00:00
Rodolfo Alonso Hernandez c686a2b555 Improve DHCP RPC handler
Remove unnecessary DB retrieval operations from "get_network_info"
method.

Partial-Bug: #1950662
Change-Id: If4b33c8437dba411fed913e7e1c7f06d899c08f7
2021-11-29 08:43:51 +00:00
Oleg Bondarev 1222962767 Add Local IP L2 extension skeleton
OVS agent part of Local IP feature was divided into
2 parts to make it easier for reviewers:

1. This patch adds agent extension skeleton and sets
server <-> agent RPC communication mechanism via
push notifications of LocalIPAssociation objects
create/delete. It also shows how the extension would
treat those changes. It may be called extension "frontend".

2. Agent extension flows patch (next one) - deals with OVS
flows and can be called extension "backend".

Partial-Bug: #1930200
Change-Id: I31cb4062b6a21b71c739ab202c60aa7002e4d36e
2021-11-11 10:25:27 +03:00
Przemyslaw Szczerbik d699a955cd Sanitize profile column of ml2_port_bindings table in the DB
With the introduction of port-resource-request-groups extension,
format of binding-profile.allocation has changed. Since the DB,
may contain port bindings that were created before the introduction
of the new format, it's necessary to perform upgrade check and
sanitize those rows that are still using an older format.

Partial-Bug: #1922237
See-Also: https://review.opendev.org/785236
Change-Id: I95e9e1bc553ac499d75c9280e45dfea61d135279
2021-10-21 15:17:24 +02:00
Zuul d94cc0d636 Merge "Execute the quota reservation removal in an isolated DB txn" 2021-10-02 03:05:26 +00:00
Rodolfo Alonso Hernandez 603abeb977 Execute the quota reservation removal in an isolated DB txn
The goal of [1] is to, in case of failing when removing the quota
reservation, continue the operation. Any expired reservation will
be removed automatically in any driver.

If the DB transaction fails, it should affect only to the reservation
trying to be deleted. This is why this patch isolates the
"remove_reservation" method and guarantees it is called outside an
active DB session. That guarantees, in case of failure, no other DB
operation will be affected.

This patch also partially reverts [2] but still checks the security
group rule quota when a new security group is created. Instead of
creating and releasing a quota reservation for the security group
rules created, now only the available quota limit is checked before
creating them. That won't prevent another operation to create security
group rules in parallel, exceeding the available quota. However, this
is not even guaranteed with the current quota driver.

[1]https://review.opendev.org/c/openstack/neutron/+/805031
[2]https://review.opendev.org/c/openstack/neutron/+/701565

Closes-Bug: #1943714

Change-Id: Id73368576a948f78a043d7cf0be16661a65626a9
2021-09-30 13:53:23 +00:00
Przemyslaw Szczerbik 56044db26d Add API extension for QoS minimum pps rule
This patch implements support for CRUD operations for QoS minimum
packet rate, for example:

DELETE /qos/policies/$POLICY_ID/minimum_packet_rate_rules/$RULE_ID

Placement or dataplane enforcement is not implemented yet.

Partial-Bug: #1922237
See-Also: https://review.opendev.org/785236
Change-Id: Ie994bdab62bab33737f25287e568519c782dea9a
2021-09-29 12:27:30 +02:00
Slawek Kaplonski e610a5eb9e Don't use singleton in routes.middleware.RoutesMiddleware
It seems that using default singleton=True in the
routes.middleware.RoutesMiddleware which is leading to use thread-local
RequestConfig singleton object is not working well with eventlet
monkeypatching of threading library which we are doing in Neutron.
As a result it leaks memory in neutron-api workers every time when API
request to not existing API endpoint is made by user.

To avoid that memory leak, let's use singletone=False in that
RoutesMiddleware object, at least until problem with thread-local
singleton and eventlet monkey patching will be solved.

Closes-Bug: #1942179
Change-Id: Id3a529248d3984506f0166bdc32e334127a01b7b
2021-09-04 09:25:19 +00:00
Nurmatov Mamatisa 92c636d8b2 Use payload callback for RPC resource_cache
This patch switches the code over to the payload style of callbacks [1].

[1]https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html

Change-Id: I2e65470e6f37ebccde01bdf3f9ed31b29567398f
2021-08-31 13:39:06 +00:00
Zuul 96f1ea140e Merge "Remove ``ConfDriver`` code" 2021-08-16 17:36:13 +00:00
Rodolfo Alonso Hernandez cbef6cda98 Add "network_id" to "_after_router_interface_deleted" payload
This parameter, sent by the DHCP agent, is needed to remove the
workaround method "_get_network_lock_id".

The removal of this method will be done in [1] in Y release.

Related-Bug: #1732456

[1]https://review.opendev.org/c/openstack/neutron/+/800967

Change-Id: Ibd7fed33d314e901c69da33f42029f8ea67df98d
2021-07-28 11:26:27 +00:00
Rodolfo Alonso Hernandez ad31c58d60 Remove ``ConfDriver`` code
The quota driver ``ConfDriver`` was deprecated in Liberty release.

``NullQuotaDriver`` is created for testing although it could be used
in production if no quota enforcement is needed. However, because
the Quota engine is not plugable (is an extension always loaded), it
could be interesting to make it plugable as any other plugin.

This patch also creates a Quota engine driver API class that should be
used in any Quota engine driver. Currently it is used in the three
in-tree drivers implemented: ``NullQuotaDriver``, ``DbQuotaDriver``
and ``DbQuotaNoLockDriver``.

Change-Id: Ib4af80e18fac52b9f68f26c84a215415e63c2822
Closes-Bug: #1928211
2021-07-26 15:00:32 +00:00
Nurmatov Mamatisa 3cae410b30 use payloads for PORT AFTER_DELETE events
This patch switches over to callback payloads for PORT
AFTER_DELETE events.
Some shims were removed.

Change-Id: If69e37b84fe1b027777b1d673b3d08a6651a979e
2021-07-11 06:00:08 +00:00
Nurmatov Mamatisa 129b823a8b use payloads for PORT AFTER_UPDATE events
This patch switches over to callback payloads for PORT
AFTER_UPDATE events.

Change-Id: I5c00eae155afa6c0fc8e3956bc39edbeca3ea1e7
2021-07-07 21:01:24 +00:00
Nurmatov Mamatisa cd8c4f7e30 use callback payloads for SUBNET
This patch switches over to callback payloads for
SUBNET events.

Change-Id: Ic4c3490aed4f899293be993d4663bb537c34ab8b
2021-06-24 00:14:52 +03:00
Slawek Kaplonski d7371e13e4 Revert "Set system_scope='all' in elevated context"
This reverts commit 062336e59b.

Now, we have proper fix for the system_scope='all' in elevated context
in the neutron-lib so we can revert temporary fix made at the end of the
Wallaby cycle.

Related-Bug: #1920001

Conflicts:
    neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py
    neutron/common/utils.py
    neutron/db/address_group_db.py
    neutron/services/segments/db.py

Change-Id: Ife9b647b403bdd76a8a99984ea8858bf95c96bc3
2021-06-15 10:29:20 +02:00
Zuul d9f12cc322 Merge "use payloads for ROUTER_INTERFACE events" 2021-06-09 15:06:23 +00:00
Nurmatov Mamatisa c9fce3a8b6 use payloads for PORT AFTER_CREATE events
This patch switches the code over to the payload style of callbacks [1]
for PORT AFTER_CREATE events. In addition it adds a branch/shim to the
dhcp_rpc_agent_api to support both payload and kwarg style callbacks.

NeutronLibImpact

[1]
https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html

Change-Id: I25d43d4f8f2390b07e0d11c631f894d88669bbe0
2021-06-03 18:43:30 +00:00
Nurmatov Mamatisa 4ab699e5cd use payloads for ROUTER_INTERFACE events
This patch switches the code over to the payload style of callbacks [1]
for ROUTER_INTERFACE events for those that are not using them yet.
The unit tests are also updated where needed to account for the
payload style callbacks and publish() method. In addition, a few
callback methods that use the retry_if_session_inactive() decorator are
separated out from the callback so that the context can still be
passed and detected by retry_if_session_inactive logic.

NeutronLibImpact

[1]
https://docs.openstack.org/neutron-lib/latest/contributor/callbacks.html

Change-Id: I8d9f8296952dfb10fcccd6afd72e90a5d4f379eb
2021-06-03 18:42:19 +00:00
Zuul 3b3398b8c5 Merge "use payloads for NETWORK callback events" 2021-05-16 08:50:19 +00:00
Zuul 9241c76b04 Merge "Add enable_dhcp filter in get_network_info rpc" 2021-05-08 22:01:59 +00:00
Nurmatov Mamatisa 4aa5de254d use payloads for NETWORK callback events
This patch switches over to the payload style of callbacks for
NETWORK based events. As part of this change a few shims are needed
to handle cases where some callbacks don't yet use payloads and others
do. Once we move over to payloads for all callbacks the shims can be
removed.

NeutronLibImpact

Change-Id: I889364b5d184d47a79fe6ed604ce13a4b334acfa
2021-05-08 20:50:46 +03:00
Nurmatov Mamatisa 1483b63ffa Add enable_dhcp filter in get_network_info rpc
Add enable_dhcp, to make a filter to avoid unnecessary
net_info data transfer through rpc.

Change-Id: Ibcef366f5b1f4b7da4f47f1f538a17111da0faa1
Closes-Bug: #1552614
2021-04-15 17:36:17 +03:00
Oleg Bondarev 130655cdb9 DHCP notification optimization
DHCP notification is done after each create/update/delete for
network, subnet and port.
This notification currently has to retrieve network from DB almost
every time, which is a quite heavy DB request and hence affects
performance of port and subnet CRUD.

This patch suggests 2 optimizations:
- do not fetch network if not needed (only fetch when schedule needed)
- for port and subnet AFTER_CREATE event pass network dict from plugin

According to Rally tests these changes improve performance:
- port create ~20%
- port update ~20%
- subnet create ~15%
- port delete and subnet update/delete - not tested

Closes-Bug: #1923161
Change-Id: I0ab836ac09225f4f3ad435e9ceaf315018855d52
2021-04-09 13:33:43 +03:00
Slawek Kaplonski 062336e59b Set system_scope='all' in elevated context
In case when enforce_new_defaults is set to True and new policy rules
are used, context.is_admin flag isn't really working as it was with old
rules.
But in case when elevated context is needed, it means that we need
context which has full rights to the system. So we should also set
"system_scope" parameter to "all" to be sure that system scope queries
can be done with such elevated context always.

It is needed e.g. when elevated context is used to get some data from
db. In such case we need to have db query which will not be scoped to
the single project_id and with new defaults to achieve that system_scope
has to be set to "all".

Proper fix for that should be done in neutron-lib and it is proposed
in [1] already but as we are have frozen neutron-lib version for
stable/wallaby already this patch for neutron is temporary fix for that
issue.
We can revert that patch as soon as we will be in Xena development cycle
and [1] will be merged and released.

[1] https://review.opendev.org/c/openstack/neutron-lib/+/781625

Related-Bug: #1920001
Change-Id: I0068c1de09f5c6fae5bb5cd0d6f26f451e701939
2021-03-19 12:05:56 +01:00
LIU Yulong 8e3a83c213 Config option to disable the DHCP functions
This patch adds a new config option ``enable_traditional_dhcp``,
if set False, neutron-server will disable:
* DHCP provisioning block
* DHCP scheduler API extension
* Network scheduling mechanism
* DHCP RPC/notification

Partially-Implements: bp/distributed-dhcp-for-ml2-ovs
Related-Bug: #1900934

Change-Id: Icfbfc9691c5cf837406ff4291b3e3ed4970b26ee
2021-03-05 14:35:29 +08:00
Hang Yang 9f09b1fb19 Support address group in OVS firewall agent
Support security group rules with remote_address_group_id in openvswitch
firewall. This change reuses most of the firewall functions handling remote
security groups to also process remote address groups. The conjunctive flows
for a rule with remote_adress_group_id are similar to others with
remote_group_id but have different conj_ids.

Change-Id: I8c69e62ba56b0d3204e9c12df3133126071b92f7
Implements: blueprint address-groups-in-sg-rules
2021-02-08 13:28:06 -06:00
Rodolfo Alonso Hernandez f3c229b9cc Process DHCP events in order if related
When processing port events (create, update, delete), the port
provisioning (port creation) has priority over the other events [1].
As reported in the related bug, if a port deletion with an IP
address and another port creation with the same IP address arrive
to the DHCP agent, those events can be processed in the same queue.

Because of the creation event priority, even when this event arrived
after the deletion event, it will be processed first. That will
clash with the DHCP agent cache, that contains a port (not deleted
yet) with the same IP address. That will trigger an unwanted resync.

This patch implements a specific logic to store the events in
"ResourceProcessingQueue" (that uses "PriorityQueue" [2]). When
a port event arrives, the event comparison method checks the
(subnet, fixed_ips) tuple set of both elements. If there is a
coincidence, that means those ports are the same or are using
the same IP addreses (the race condition explained in the bug).
In this case, the priority is defined only by the timestamp;
that means the events are processed in order of arrival.

Because the Neutron server do not allow to have two ports in the
same subnet with the same IP address, the order of the events is
guaranteed. In the case explained in the bug, the deletion event
will be processed first.

[1]https://review.opendev.org/c/openstack/neutron/+/626830
[2]https://docs.python.org/3/library/queue.html#queue.PriorityQueue

Closes-Bug: #1913723

Change-Id: I89438feae3c0244f6da5e6a2a035d45b956ac247
2021-02-04 15:39:10 +00:00
Zuul f84d2f74f5 Merge "Get sec group ids after address group update" 2021-01-09 02:00:45 +00:00