This method populates the SG rules in a dictionary. Each SG rule
inherits the "stateful" value of the SG. Prior to this patch, each
SG rule was isuing a database call to retrieve the SG register.
In this patch, the SG "stateful" retrieval is done in one database
query for all SG. That improves the performance of this method
reducing the database access to only one single call.
This improvement, as commented in the LP bug, affects to
ML2/LinuxBridge. ML2/OVS agent uses a cached RPC implementation
that not requires to perform any RPC call/database query.
Closes-Bug: #2045950
Change-Id: Iafd0419a1d1eeb25d5589edc2570ebf287450957
This makes _clear_child_sg_rules look into the local resource cache
for sg rules that have to be deleted instead of querying the
neutron-rpc server.
Deleting a security group results in each agent of the region running
a bulk_pull query for all the rules in the security group against the
neutron-rpc. This is incurs a load on neutron-rpc, rabbitmq and the db
proportional to the number of agents and the number of security group
rules and has a noticeable impact on larger infrastructures.
However, when the cache is properly populated, we already have the
rules we should remove from the cache and don't need to ask neutron
server for them.
Closes-Bug: #2022360
Change-Id: I53e11f558a6cd84a02041758badddfe87a10c95c
rpc_workers can be set < 1 with 'ovn' backend when no
other agent is running apart from ovn agents to
consume these rpc notifications.
Add and apply disable_notifications decorator on
methods which do rpc cast calls to agents, the
decorator makes the caller method execute only
when rpc_workers >=1. This patch not changing
default behavior and utilizes the rpc_workers config option
to enable rpc notification on resources updates only when
rpc_workers >= 1.
Also set rpc_workers=0 in ovn jobs to cover this scenario.
Closes-Bug: #1889737
Closes-Bug: #1992352
Change-Id: I700fe2cd422bc1eb8b5144ec116e7f0a60238419
Support security group rules with remote_address_group_id in openvswitch
firewall. This change reuses most of the firewall functions handling remote
security groups to also process remote address groups. The conjunctive flows
for a rule with remote_adress_group_id are similar to others with
remote_group_id but have different conj_ids.
Change-Id: I8c69e62ba56b0d3204e9c12df3133126071b92f7
Implements: blueprint address-groups-in-sg-rules
This change adds code to retrieve for the agent the security group ids
affected by an update or deletion of an address group.
Also adds event notificatoins to add and remove addresses from address
groups.
Co-authored-by: Hang Yang <hangyang@verizonmedia.com>
Change-Id: I34766b96cb775356664f5e0d48a08a22ac6898e2
A partial upgrading of neutron cluster, neutron-server
has a newer version while neutron-agents not, does not
run well after a RPC data structure upgrading. This
patch upgrades the security group related RPC version
between neutron-server and agents. A partial upgrading
neutron cluster will explicitly raise error. The RPC
version should be aligned.
Closes-bug: #1903531
Related-bug: #1867119
Change-Id: I6cb2ba05fa3337be46eb01f2d9f869efa41e4db6
Adds agent side code to enable the OVS agent to receive address groups
from the push notifications cache.
Change-Id: I1f27eccb2a69c553631fdc12d34e9025925844c5
Partial-Bug: #1592028
As spotted in Focal testing patch [0], pep8 test fails with many
C0321 false-positives, reported in pylint as current version does not
support python 3.8 [1]
Use a newer version of pylint and astroid, fixing or disabling some of
the new checks: no-else-*, unnecessary-comprehension, import-outside-toplevel
[0] https://review.opendev.org/#/c/738163/
[1] https://github.com/PyCQA/pylint/issues/2737
Change-Id: Ie646b7093aa8634fd950c136a0eba9adcf56591c
When add allowed-address-pair 0.0.0.0/0 to one port, it will
unexpectedly open all others' protocol under same security
group. IPv6 has the same problem.
The root cause is the openflow rules calculation of the
security group, it will unexpectedly allow all IP(4&6)
traffic to get through.
For openvswitch openflow firewall, this patch adds a source
mac address match for the allowed-address-pair which has
prefix lenght 0, that means all ethernet packets from this
mac will be accepted. It exactly will meet the request of
accepting any IP address from the configured VM.
Test result shows that the remote security group and
allowed address pair works:
1. Port has 0.0.0.0/0 allowed-address-pair clould send any
IP (src) packet out.
2. Port has x.x.x.x/y allowed-address-pair could be accepted
for those VMs under same security group.
3. Ports under same network can reach each other (remote
security group).
4. Protocol port number could be accessed only when there
has related rule.
Closes-bug: #1867119
Change-Id: I2e3aa7c400d7bb17cc117b65faaa160b41013dde
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
NeutronLibImpact
Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.
NeutronLibImpact
Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
Neutron lib contains the latest callbacks and thus this patch removes
the callbacks package from neutron entirely.
NeutronLibImpact
Change-Id: I14e45fd5d2d3c816bb39f8ace56f7be460bac0d6
Changing rpc_api.rst file path from doc/source/devref/rpc_api.rst
to /doc/source/contributor/internals/rpc_api.rst. Because rpc_api.rst
file is located at this path
doc/source/contributor/internals/rpc_api.rst.
Closes-Bug #1722072
Change-Id: Ic243aab9e3428bfec69db61a94b4129cd768e233
In Pike, the agent side of security_groups_provider_updated()
RPC code was changed to a NOOP when the provider rules were
changed to be static, https://review.openstack.org/#/c/432506
Now that we're in Queens we can deprecate it.
Change-Id: Ie018ff653633d3524f0e80c5e172a5d01bdad437
SecurityGroupServerRpcApi and SecurityGroupServerRpcCallback are a pair,
and SecurityGroupAgentRpcApiMixin and SecurityGroupAgentRpcCallbackMixin
should be a pair.
Change-Id: I12fe4beecbbcb222b84aacf262aa9c76039ff9ce
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
Firewall drivers check if port security is enabled. After ovo is sent
over the wire, the port_security_enabled is part of 'security' field.
The patch translates the RPC call from agent to server so the payload
containing port_security_enabled is at the same place.
We may consider implementing change of OVO field to contain boolean
directly.
Change-Id: I647343e84b41da63d7ffcc5a87f3dfa2036adc56
Closes-bug: #1605654
Calculate all security group info on the agent from
the push notification cache.
Partially-Implements: blueprint push-notifications
Change-Id: I5c74ba17223a431dad924d31bbe08ad958de3877
neutron-lib 1.6.0 is out and among other things contains the rehomed
is_port_trusted function. This patch switches usage of that function
from neutron to neutron-lib.
NeutronLibImpact
Change-Id: I1c8a32d4806092daae99cd4860523c7d6335fc75
Setting up rules to allow DHCPv6, DHCP, and RAs from specific
IP addresses based on Neutron resources has a few issues:
1. It violates separation of concerns. We are implementing logic to
calculate where an IPv6 RA advertisement or DHCP advertisement
should be coming from in the security group code. This code should
not be trying to guess IPv6 LLAs, know about subnet modes, DHCP server
implementations, or the type of L3 plugin being used. Currently all
of these assumptions are baked into code that should only be
filtering, which makes it very rigid and brittle when it comes to
other implementations for DHCP and/or RAs.
2. It has scaling issues on large networks. Every time one of these
provider rules is updated, it triggers every L2 agent to refresh
all of the security group rules for ports in that network, which puts
significant load on the server.
3. It's main purpose: preventing spoofing of RA[1,2] and DHCP packets,
has long been superceded by preventing VMs from acting as DHCP/RA
servers[3][4].
This patch completely removes all of this logic and just returns
static provider rules to the agents that allow all DHCP server
and RA traffic ingress to the client. This addresses the issues
highlighted above since the code is significantly simplified and
the provider rules don't require refreshes on the agents.
Now that the provider rules never change, the RPC notification
listener on the agent-side for 'notify_provider_updated' is now
just a NOOP that doesn't trigger any refreshes. The notification
was left in place on the server side for older version agents
that have stale IP-specific provider rules. The entire notification
can be removed in the future.
The one open concern with this approach is that VMs will now be
able to receive DHCP offers from other DHCP servers on the same
network that aren't being filtered (e.g. a VM with port security
disabled or another device on a provider network). In order to
address this for DHCP, this patch adds two rules that only allow
DHCP offers targeted to either the broadcast or the correct client
IP. This prevents incorrect offers from ever reaching the client.
For RAs, this patch just allows all RAs so we may pick up
advertisements from other v6 routers attached to a network;
however, the instance won't actually be allowed to use bad addresses.
1. https://bugs.launchpad.net/neutron/+bug/1262759
2. I1d5c7aaa8e4cf057204eb746c0faab2c70409a94
3. Ice1c9dd349864da28806c5053e38ef86f43b7771
4. https://git.openstack.org/cgit/openstack/neutron/tree/
neutron/agent/linux/iptables_firewall.py
?h=521b1074f17574a5234843bce68f3810995e0e1d#n475
Closes-Bug: #1653830
Closes-Bug: #1663077
Change-Id: Ibfbf011284cbde396f74db9d982993f994082731
Neutron Manager is loaded at the very startup of the neutron
server process and with it plugins are loaded and stored for
lookup purposes as their references are widely used across the
entire neutron codebase.
Rather than holding these references directly in NeutronManager
this patch refactors the code so that these references are held
by a plugin directory.
This allows subprojects and other parts of the Neutron codebase
to use the directory in lieu of the manager. The result is a
leaner, cleaner, and more decoupled code.
Usage pattern [1,2] can be translated to [3,4] respectively.
[1] manager.NeutronManager.get_service_plugins()[FOO]
[2] manager.NeutronManager.get_plugin()
[3] directory.get_plugin(FOO)
[4] directory.get_plugin()
The more entangled part is in the neutron unit tests, where the
use of the manager can be simplified as mocking is typically
replaced by a call to the directory add_plugin() method. This is
safe as each test case gets its own copy of the plugin directory.
That said, unit tests that look more like API tests and that rely on
the entire plugin machinery, need some tweaking to avoid stumbling
into plugin loading failures.
Due to the massive use of the manager, deprecation warnings are
considered impractical as they cause logs to bloat out of proportion.
Follow-up patches that show how to adopt the directory in neutron
subprojects are tagged with topic:plugin-directory.
NeutronLibImpact
Partially-implements: blueprint neutron-lib
Change-Id: I7331e914234c5f0b7abe836604fdd7e4067551cf
Initially, we bumped the required version for the agent endpoint from
1.1 (the initial version that implemented security groups) to 1.3
without considering that the code should work with old agents that do
not yet know about the new devices_to_update argument.
Actually, there was no need to bump the version: old agent side code
already captures all unknown arguments that could be passed from the
server, ignoring them:
608b54137f/neutron/agent/securitygroups_rpc.py (L155)
(^ the link shows the latest Kilo code as of writing)
Note: some people may argue that the approach that is taken in Neutron
to support backwards compatibility for server notifications is wrong,
and we instead should adopt some stricter mechanism like nova version
pinning. While that is a noble thing to do, it's out of scope for the
patch that is designed to be easily backportable to stable/liberty.
Note: some people may also argue that the patch should go straight into
stable/liberty because we don't claim support for rolling upgrade
scenarios that span multiple releases. That's indeed true, though my
take on it is that if we have a way to handle more unofficial scenarios
without more coding effort, it's worth doing it.
Change-Id: I741e6e5c460658ac17095551040e67e8d1990812
Closes-Bug: #1531772
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
Ports that have a device_owner that starts with 'network:'
are trusted in several places throughout the codebase. Each
of these did a startswith check on each field and it's not
immediately obvious why it's done.
This patch adds a utility function called 'is_port_trusted'
that performs the same check and makes it obvious what is
being done.
Change-Id: I542c753776d5cfb2fd736b25ea6e111867c89c89
This patch changes the get_port_from_device_mac() and
get_sg_ids_grouped_by_port() methods in ML2 db.py module so that
they do not create a new database session (via get_session()), but
instead reuse the session associated with the caller's context.
In order to make the session that is associated with the caller's
context available to these ML2 DB methods, the
get_ports_from_devices plugin API in securitygroups_rps_base.py
needs to be modified so that the context can be passed down to the
ML2 plugin. (A similar change is made to the get_port_from_device
plugin API for consistency.)
Change-Id: I3f990895887e156de929bd7ac3732df114dd4a4b
Closes-Bug: 1441205
When a security_groups_provider_updated is received then a global
refresh of the firewall is performed. This can be avoided if the
plugins pass as parameter of the call the devices that belongs to
the network updated.
Partially-Implements: blueprint restructure-l2-agent
Change-Id: I1e78f3a5ec7e5c5bcba338a0097566422411ef7e
Use set operations instead of using list.
Currently complexity of the method is O(n^2) where n is
amount of ips (amount of VMs in the network).
When amount of VM is big (large L2 domain), this method
can significantly load the controller.
Reduce method complexity to O(n) on average.
Change-Id: If1660e8227e5c5cd80d49ebcc6a2e06d33d31939
Closes-Bug: #1429753
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
This patch scopes the agent to plugin security group rpc interface
using a messaging namespace. Right now some plugins expose several
interfaces via the default namespace. This effectively means they are
a single API and should be managed with a single version stream. It's
much more managable to just treat these as separate interfaces and
this change makes that explicit and functionally true. Now when a
method is invoked, the only classes considered for handling that
request will be ones marked with the right namespace.
Part of blueprint rpc-docs-and-namespaces.
Change-Id: Iaee934646c9da7d32968406a583a5718fffc893b
Create SecurityGroupServerRpcApi, which is intended to replace
SecurityGroupServerRpcApiMixin over time. The new class is the same
code, but used as a standalone class. This is needed to be able to
eventually move this API into a messaging namespace.
In passing, add docstrings that show where to find the corresponding
other side of this interface in the code base.
Part of blueprint rpc-docs-and-namespaces.
Change-Id: If959b2f8bf21b4963bf6876a3eedc0a4b06c6400
This patch drops several uses of the RpcCallback compatibility class.
All of these were trivial and straight forward conversions so I
batched them up.
There are still several other uses of RpcCallback, but the conversions
are were not necessarily trivial, and may be broken in one way or
another, so I wanted to address them separately. In particular, the
use of mixin classes means that there could be cases where the version
declaration is being stepped on, so they need to be investigated more
closely.
Part of blueprint drop-rpc-compat.
Change-Id: I0977aee863d4fcc6a14a025215c6e41fe38a87f0
The security groups RPC handler calls get_port_from_device
individually for each device in a list it receives. Each
one of these results in a separate SQL query for the security
groups and port details. This becomes very inefficient as the
number of devices on a single node increases.
This patch adds logic to the RPC handler to see if the core
plugin has a method to lookup all of the device IDs at once.
If so, it uses that method, otherwise it continues as normal.
The ML2 plugin is modified to include the batch function, which
uses one SQL query regardless of the number of devices.
Closes-Bug: #1374556
Change-Id: I15d19c22e8c44577db190309b6636a3251a9c66a
Follow-up patch of RPC refactoring of bug 1359416.
It addresses minor comments in the above patch series.
Change-Id: I2d6268db777f0f73fda61a5a7d0967a91bcb292b
Closes-Bug: #1359416
Refactor rpc call of 'security_group_rules_for_devices' into
'security_group_info_for_devices' to reduce the response
message size and server side processing time. Includes
agent-side auto detection of rpc version, and fall back
to the old one.
Change-Id: If19be8579ca734a899cdd673c919eee8165aaa0e
Implements: blueprint security-group-rules-for-devices-rpc-call-refactor
DocImpact
Co-Authored-By: Miguel Angel Ajo <mangelajo@redhat.com>
RPC has a version of itself. In Neutron a plugin implements
several RPC interface, so a single RPC version doesn't work.
In Mixin callback class approach, RPC versioning depends on
each plugin implementation and it makes harder to maintain
RPC version appropriately. This patch series replaces mixin
RPC callback of server side with a separate class.
This commit handles server-side callback of security group
RPC interface.
* The server-side callback of Security group RPC is moved to
api/rpc/handler and db/securitygroups_rpc_base now only
contains a mixin class to add agent-based security group
implementation with db operations.
* get_port_from_device method in server-side callback class
is moved to a mixin class of plugin implementation
(SecurityGroupServerRpcMixin) because it involves DB lookup
and is tightly coupled with plugin implementation rather
than RPC interface definition.
Most unit tests for SGServerRpcCallBackTestCase were skipped
in the base class before, but now they are no longer skipped.
The following items will be planned in later patches
to avoid drastic changes in a single patch.
* Merge security group RPC API and agent callback classes in
agent/securitygroups_rpc into api/rpc/handlers/securitygroup_rpc
* Remove completely duplicated db access code in get_port_from_device
and get_port_and_sgs
Partial-Bug: #1359416
Change-Id: Ia6535217d2e3b849a95667c1b53dd09675002892
Rodolfo Alonso Hernandez
yatinkarel