Importing some modules lead to registering config options that may
collide with config options from a project that calls the import. This
patch wraps the side effect that registers config options into a
function that needs to be called in case the caller wants to register
the options.
This solution is also not perfect as it guards the common options to be
registered only once even if the function is called multiple times. This
is to solve problems in unittests, ideally we should always call the
function just once even in our testing suites.
Resolves-Bug: #1968606
Change-Id: Ic1532eb8de887ff1b1085206df11f53e22f7f524
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Now that we use setproctitle for neutron-server workers (and
neutron-keepalived-state-change), this has the side effect of changing
the process name for agents, impacting some monitoring systems. More
details in launchpad bug.
This patch fixes it by setting the name with setproctitle to:
agent name (original process name).
Also use the newly introduced name constants to replace existing
hardcoded uses.
Change-Id: I74c3a4d3e9f833752571a75f196560cd45529385
Closes-Bug: #1881297
Remove this duplicated option and rely only in OVS.integration_bridge.
NOTE: other projects are still using it; first we need to deprecate it
in those projects.
Change-Id: I4e826c8b9fa764b1820adacc3427934dc393c0bc
Related-Bug: #1856152
New releases of oslo.config support a 'mutable' parameter to Opts.
This is only respected when the new method mutate_config_files is
called instead of reload_config_files. Neutron delegates making this
call to oslo.service. This was provided in patchset
Icec3e664f3fe72614e373b2938e8dee53cf8bc5e
Further patches will be needed to make select config options be
marked as mutable. This change enables support for oslo provided
config options to be updated via SIGHUP such as log level.
Task: 6389
Story: 2001545
Change-Id: I9442965607f3248706464643c6d87a04edcae24e
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.
NeutronLibImpact
Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
Change network namespace add/delete/list code to use
pyroute2 library instead of calling /sbin/ip.
Also changed all in-tree callers to use the new calls.
Closes-bug: #1717582
Related-bug: #1492714
Change-Id: Id802e77543177fbb95ff15c2c7361172e8824633
Refactoring neutron agent linux and ovsdb config opts
to be in neutron/conf/agent so that all the config options
reside in a centralized location. This simplifies the
process of looking up the config opts and provides an easy
way to import.
NeutronLibImpact
Change-Id: Ib1e0e63dec2985c417412d1ecc68e2a74ef87182
Partial-Bug: #1563069
Refactoring Neutron configuration options for agent common config to be
in neutron/conf/agent/common. This will allow centralization of all
configuration options and provide an easy way to import.
Partial-Bug: #1563069
Change-Id: Iebac0cdd3bcfd0135349128921b7ad7a1a939ab8
Needed-By: Ib676003bbe909b5a9013a3178b12dbe291d936af
Due to the high memory footprint of current Python ns-metadata-proxy,
it has to be replaced with a lighter process to avoid OOM conditions in
large environments.
This patch spawns haproxy through a process monitor using a pidfile.
This allows tracking the process and respawn it if necessary as it was
done before. Also, it implements an upgrade path which consists of
detecting any running Python instance of ns-metadata-proxy and
replacing them by haproxy. Therefore, upgrades will take place by
simply restarting neutron-l3-agent and neutron-dhcp-agent.
According to /proc/<pid>/smaps, memory footprint goes down from ~50MB
to ~1.5MB.
Also, haproxy is added to bindep in order to ensure that it's installed.
UpgradeImpact
Depends-On: I36a5531cacc21c0d4bb7f20d4bec6da65d04c262
Depends-On: Ia37368a7ff38ea48c683a7bad76f87697e194b04
Closes-Bug: #1524916
Change-Id: I5a75cc582dca48defafb440207d10e2f7b4f218b
Refactoring neutron agent metadata config opts to be in
neutron/conf/agent/metadata so that all the configurations options
reside in a centralized location. This simplifies the process of looking
up the config opts and provides an easy way to import.
Change-Id: I8bae1facc58a4f9e21196f625478532403651545
Partial-Bug: #1563069
Refactoring neutron configuration options for dhcp agent to be in
neutron/conf/agent. This would allow centralization of all configuration
options and provide an easy way to import.
Change-Id: Ia17d2d7223dd598e2d36a8320942fb03b61dffaf
Partial-Bug: #1563069
The use_namespaces option has been defined as a workaround to kernels
not properly supporting namespaces. This limitation is behind us, it's
time to remove use_namespaces after its deprecation in Kilo in order to
simplify code and remove a poorly tested case (use_namespaces=False).
This change prepares for removal pullup_route method[1] which was only
used when use_namespaces=False.
[1] neutron.agent.linux.ip_lib
DocImpact
UpgradeImpact
Closes-Bug: #1508188
Related-Bug: #1435382
Depends-On: I303038eec560a6d99421140c2822aed8b518470b
Depends-On: I4feb2a15c7e1e4bfdbed2531b18b8e7d798ab3cc
Change-Id: I2fbf65df1250d9f9f1656b3964ee3b6de1ef1118
This patch adds the availability_zone attribute to agents and
supports availability_zone API.
Availability_zone support for resources (network/router) and
the schedulers are included in subsequent patches.
APIImpact
DocImpact
Co-Authored-By: IWAMOTO Toshihiro <iwamoto@valinux.co.jp>
Change-Id: Id7a62000ab0484412b3970199df8c374568fe70d
Partially-implements: blueprint add-availability-zone
This patch doesn't changes behaviour of dhcp-agent
but adds the opportunity to use user-defined config,
that will make dhcp-agent more flexible
and allows to run functional tests correctly
(without changing global oslo.config CONF)
Closes-Bug: #1492283
Change-Id: Ice807e8fc872b56bb3960b7a3de4110c7675d9d6
oslo.service has graduated, so neutron should consume it.
Closes-Bug: #1466851
Depends-On: Ie0fd63f969f954029c3c3cf31337fbe38f59331a
Depends-On: I2093b37d411df9a26958fa50ff523c258bbe06ec
Depends-On: I4823d344878fc97e66ddd8fdae25c13a34dede40
Change-Id: I0155b3d8b72f6d031bf6f855488f80acebfc25d4
Currently metadata proxy cannot run with nobody user/group as metadata
proxy requires to connect to metadata_proxy_socket when queried.
This change allows to run metadata proxy with nobody user/group by
allowing to choose the metadata_proxy_socket mode with the new option
metadata_proxy_socket_mode (4 choices) in order to adapt socket
permissions to metadata proxy user/group.
This change refactors also where options are defined to enable
metadata_proxy_user/group options in the metadata agent.
In practice:
* if metadata_proxy_user is agent effective user or root, then:
* metadata proxy is allowed to use rootwrap (unsecure)
* set metadata_proxy_socket_mode = user (0o644)
* else if metadata_proxy_group is agent effective group, then:
* metadata proxy is not allowed to use rootwrap (secure)
* set metadata_proxy_socket_mode = group (0o664)
* set metadata_proxy_log_watch = false
* else:
* metadata proxy has lowest permissions (securest) but metadata proxy
socket can be opened by everyone
* set metadata_proxy_socket_mode = all (0o666)
* set metadata_proxy_log_watch = false
An alternative is to set metadata_proxy_socket_mode = deduce, in such
case metadata agent uses previous rules to choose the correct mode.
DocImpact
Closes-Bug: #1427228
Change-Id: I235a0cc4f0cbd55ae4ec1570daf2ebbb6a72441d
Currently neutron-ns-metadata-proxy runs with root permissions when
namespaces are enabled on the dhcp agent because root permissions are
required to "enter" in the namespace. But neutron-ns-metadata-proxy
permissions should be reduced as much as possible because it is
reachable from vms.
This change allows to change neutron-ns-metadata-proxy permissions
after its startup through the 2 new options metadata_proxy_user and
metadata_proxy_group which allow to define user/group running metadata
proxy after its initialization. Their default values are
neutron-dhcp-agent effective user and group.
This change delegates metadata proxy management to metadata driver
methods in order to reuse the work already done on l3 agent side.
Permissions drop is done after metadata proxy daemon writes its
pid in its pidfile (it could be disallowed after permissions drop) and
after metadata proxy daemon binds its privileged server port (80).
Using nobody as metadata_proxy_user/group (more secure) is currently
not supported because:
* nobody has not the permission to connect the metadata socket,
* nobody has not the permission to log to file because neutron uses
WatchedFileHandler (which requires read/write permissions after
permissions drop).
This limitation will be addressed in a daughter change.
DocImpact
Closes-Bug: #1187107
Change-Id: I53e97254d560e608101010f67bd2dcdec81fb6a2
Oslo project decided to move away from using oslo.* namespace for all their
libraries [1], so we should migrate to new import path.
This patch applies new paths for:
- oslo.config
- oslo.db
- oslo.i18n
- oslo.messaging
- oslo.middleware
- oslo.rootwrap
- oslo.serialization
- oslo.utils
Added hacking check to enforce new import paths for all oslo libraries.
Updated setup.cfg entry points.
We'll cleanup old imports from oslo-incubator modules on demand or
if/when oslo officially deprecates old namespace in one of the next
cycles.
[1]: https://blueprints.launchpad.net/oslo-incubator/+spec/drop-namespace-packages
Depends-On: https://review.openstack.org/#/c/147248/
Depends-On: https://review.openstack.org/#/c/152292/
Depends-On: https://review.openstack.org/#/c/147240/
Closes-Bug: #1409733
Change-Id: If0dce29a0980206ace9866112be529436194d47e
Wrap dhcp agent into its own module and break out configurations and
entry point for better seperation. This lead to some test cleanup
that revealed that options were registered unnecessarily.
When/if the dhcp agent goes through a restructuring along the same
lines of the L3 agent's, this would be the step to start from.
Related-blueprint: restructure-l3-agent
Related-blueprint: core-vendor-decomposition
Change-Id: I87d9f1079ed4e71c731984ec00e2f785024fd5f8
There is an rpc interface defined for the Neutron plugin to be able to
execute methods in the DHCP agent. Provide docstring pointers in the
client and server side that tells you where to find the other side of
the interface.
No namespace usage is needed here. This API is the only one exposed
via the DHCP agent, so the default namespace used now is fine.
The DhcpAgent class was updated to explicitly define the
messaging.Target(). Previously it was using the equivalent one
defined in the Manager base class. Having it specified here makes it
more obvious that this is an rpc endpoint, and also provides the
obvious place that must have the version updated if the interface is
changed.
Part of blueprint rpc-docs-and-namespaces.
Change-Id: I4a6eb8dacb9ba01f329a5d5961dc0e0ee6f780ba
There are a small number of examples of "eager" interpolation in
neutron:
logging.debug("foo %s" % arg)
These should be converted to perform the interpolation lazily within
the logging function, since if the severity is below the logging level
then the interpolation can be skipped entirely.
This change addresses all such current examples found in neutron core
via a pylint test. Vendor plugins and services are fixed elsewhere.
Change-Id: I823d8453cd76e4985cabd31ca6b939f43a80b36c
Partial-Bug: #1404788
This patch updates the rpc API used by the DHCP agent to make calls
back into the neutron plugin to use the 'dhcp' namespace instead of
the default namespace. The reason is that this API is exposed over
the 'q-plugin' topic along with several other interfaces. Without the
use of namespaces, all of the interfaces are effectively treated as
one by oslo.messaging. When a namespace is used, the interface can be
versioned independently and when a method is called, the only class
considered for fulfilling the request is the one that claims to
implement the 'dhcp' namespace.
While we're here, add documentation to both the client and server side
of this interface that indicates where the other side is located.
Part of blueprint rpc-docs-and-namespaces.
Change-Id: I9e56aa34fc560ae3fc749c51788436e32179d0a1
The following patch: 9569b2fe broke the desired functionality of
the enable_metadata_network flag, by not allowing the metadata
proxy to be spawn for 'metadata networks', which are used for
accessing the metadata service when the logical router is not
implemented through the l3 agent.
This patch enables spawning of the metadata proxy for metadata
networks when the appropriate flag is set to True.
The patch also adds rather pedant unit test coverage for the
should_enable_metadata method which previously had no unit test.
Change-Id: I8dca1fce9fbc83e75ba7e4ce948531427bf7e88b
Closes-bug: 1394020
Mostly trivial import changes.
- oslo.i18n no longer provide install() method to inject _() into
globals(), so removed all calls to it;
- removed Babel from dependencies (it will now be grabbed by oslo.i18n);
- updated tox.ini to ignore import violations for oslo.i18n.
Change-Id: I6623d551f512fb7fe9bf35ee734ed6d4c6cbc287
Commit 1b66e11b5d introduces
a new problem that metadata proxy will not be started when
an isolated network contains ipv6 subnet with dhcp enabled
and a ipv4 subnet.
This fix separates the checks to enable dhcp and enable metadata
so metadata proxy can be started if there is ipv4 subnet exists
in the network with metadata enabled.
Change-Id: If26806083db225e9ab30d75fa6c728ce99a7cafe
Closes-Bug: 1392564
The dhcp rpc API has two version (1.0 and 1.1). The proper way to use
versioning for this is to only specify '1.1' from the client side when
you require that the remote side implements at least version '1.1' for
the method to work. Otherwise, '1.0' should still be specified. The
previous code specified '1.1' always.
Related to blueprint drop-rpc-compat.
Change-Id: I9468f8f67d80c5d064137f917fc04f9335a3ed55
Drop the usage of the RpcProxy compatibility class from the
DhcpPluginApi RPC client class. The implementation has been updated
to use the appropariate APIs from oslo.messaging directly.
Part of blueprint drop-rpc-compat.
Change-Id: I81bfd761707c4c587b12877668c4399efe3d652e
Don't translate debug level logs and enforce log hints
Our translation policy
(https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation) calls
for not translating debug level logs. This is to help prioritize log
translation. Furthermore translation has a performance overhead, even if
the log isn't used (since neutron doesn't support lazy translation yet).
NOTE: this is done on a directory by directory basis to ensure that we
do not have too many conflicts and rebases.
Add a local hacking rule to enforce this.
This patch set enforces the directory neutron/agents
Partial-bug: #1320867
Change-Id: I4bd562e5138c2d2850072440aa121f27e902463a
Remove RemoteError from neutron.common.rpc, which was just mapped
directly to oslo.messaging.RemoteError.
Part of blueprint drop-rpc-compat.
Change-Id: Ic4d7ea0adfd68eb56100e32ce0d8e59aa2aae5b1
The previous implementation will loop through and restart the dhcp
process for all active networks any time there is an exception calling
a dhcp driver function. This allows a tenant who can create an exception
to cause every dhcp process to restart. On systems with lots of networks
this can easily take longer than the default resync timeout leading to a
system that becomes unresponsive because of the load continually restarting
causes.
This patch restarts only dhcp processes related to the network on which
operations are failing. It should be noted that if there was some kind
of missed notification for a subnet update, the previous implementation
may have incidentally fixed it by restarting everything on the off
chance that something else caused an exception, but obviously relying
on that would be a bad idea as exceptions should be, well, exceptional.
Closes-bug: #1384402
Change-Id: I0b348a1657a7eb3a595f9bf6b217716a37ce38c6
There is a bug in dhcp agent code that when first port is created
in an IPv6 subnet, DHCP port is not automatically created.
This fix resolves this problem by removing the IP version check
in configure_dhcp_for_network method.
Change-Id: If3f405d367a7099d9f33d72d11ffcb7a393abe23
Closes-Bug: #1367500
The cfg.CFG argument is ignored and misleading. This change removes it
and updates all callers.
Closes-Bug: #1355565
Change-Id: I2fcece85d1abe848e5c01125cfb62c02f2dcbc86
If the configuation option "enable_isolated_metadata = True" for the
DHCP agent is set, the neutron-ns-metadata-proxy process is spawned
for all networks, regardless if they are isolated or not. In case
the network is not isolated (ie. connected to a neutron router), the
L3 agent also spawns a proxy process, and the DHCP's proxy is left
unused. This patch adds a check prior to the spawning of new proxies:
if a network is not isolated, no proxy is spawned.
Change-Id: I9bdb8c3d37997b22435bca33ec47a67db08efa51
Closes-bug: #1361545
RpcProxy sets by default topic=self.topic, there's no need to specify
it explicitly in derived class, unless it is overridden
Change-Id: I19b9a67072a7f3c42e3b0e4ba412241a056a79a3
Closes-bug: 1348180
eventlet pool will discards all exception raised inside spawn(_n), it is
hard to discover problems inside spawned green-threads, it is better to
add a function wrapper to log the exceptions raised inside spawned
function.
Change-Id: I753df36c4614759f49134667a55f2a91f0c08317
Closes-bug: #1340708
Most of this code will probably stay with us for quite some time, so
let's make things easier and consider them as our way of doing RPC.
blueprint oslo-messaging
Change-Id: Iaf353b23f9c54b82d1e02a6bd5a5960cec827c88
After oslo.messaging port is done, we'll need to initialize RPC layer
from all RPC agents. We'll put initialization into init() function, so
the first step for migration is to make agents use it.
The function is renamed to be explicit about the fact that we don't just
parse configuration by calling it, but also do other common
initializations, like setting RPC layer.
blueprint oslo-messaging
Change-Id: I71c62f270ac7a1ff1426a3f49a32133b65580e35
In oslo.messaging, local thread store is used to pass RPC request
context [1]. If we try to import oslo.messaging before monkey patching
threading library, it attempts to access unpatched storage and fails
with AttributeError.
[1]: oslo/messaging/localcontext.py#L26
blueprint oslo-messaging
Change-Id: Ied7302fcb1d3e14428540e39e3db704550027890
These exception types will ease transition from oslo-rpc to
oslo.messaging.
blueprint oslo-messaging
Change-Id: I48d214d676e863d7a60d78c49d2fa917d47c07d9
This class is intended for easy and iterative migration from
oslo-incubator RPC implementation to oslo.messaging. It will adapt
oslo.messaging API to old oslo-rpc interfaces.
Migrated all affected classes to the new RpcProxy class.
blueprint oslo-messaging
Change-Id: I1563c13b470b61eebc982ee81efee47732943d48
Jakub Libosvar