Add config options for neutron-keepalived-state-change python bin
daemon. And it will be possible to not run root neutron-rootwrap-daemon
for it after fix of
https://review.opendev.org/c/openstack/neutron/+/653378
Related-Bug: #2052367
Change-Id: I15060c9c6b1c2c11b3a7d692f945e27ffec9e854
This patch removes the config option "debug" override done
during the script initialization.
Closes-Bug: #2003534
Change-Id: I403d73a1f35cb6314c814f25628a83d3e111e0fe
Importing some modules lead to registering config options that may
collide with config options from a project that calls the import. This
patch wraps the side effect that registers config options into a
function that needs to be called in case the caller wants to register
the options.
This solution is also not perfect as it guards the common options to be
registered only once even if the function is called multiple times. This
is to solve problems in unittests, ideally we should always call the
function just once even in our testing suites.
Resolves-Bug: #1968606
Change-Id: Ic1532eb8de887ff1b1085206df11f53e22f7f524
Signed-off-by: Jakub Libosvar <libosvar@redhat.com>
Neutron-keepalived-state-change-monitor process is using HTTP to notify
L3 agent that state of the HA router was changed. It is done through
unix socket.
With HTTP 1.1 connection isn't closed properly and if
router's state was changed more than once in short time, it could happen
that wsgi worker on the L3 agent's side didn't process second request at
all. That caused problem with transitioning router to master state after
creation.
Request to close connection should be explicitly defined in the header
[1] and this patch adds "connection: close" header to the headers send in
such request to do exactly that.
[1] https://www.geeksforgeeks.org/http-headers-connection/
Closes-Bug: #1923633
Change-Id: Ic08ca5b167db4884efa07112cf9a6c3637e1b827
The initial router state method is now a thread that is executed
in parallel with the "ip monitor" thread. If by any circumstance
this thread does not read the interface IP addresses on time, the
state of the router will be defined as "backup".
Related-Bug: #1917793
Change-Id: If76c6ee1734f544abdd4196431351d4328ad26fd
This method was intended to check state of the HA router on the
node and update it in the neutron server.
Patch [1] added check of the initial status to the
neutron_keepalived_state_change_monitor process.
It also could cause some race conditions and event which is setting
correct state of the router will be not processed thus router may endup
with two nodes with "primary" state in the Neutron's DB.
Neutron_keepalived_state_change_monitor was notifying agent about
router's initial state only if this state was 'primary'.
Now it will notify agent always to let agent set router's state as
'backup' if needed (that was previously done by this removed
update_initial_state() method).
[1] https://review.opendev.org/c/openstack/neutron/+/642295
Change-Id: I2cc58c30cf844ee0ecf0611ecdec430086464790
Closes-Bug: #1916022
Previously, to work around a bug in keepalived, the
neutron-keepalived-state-change-monitor process was altered to emit
garps under certain circumstances. That issue with keepalived issue
has been resolved, so remove the extraneous garp.
Two tests that tested the removed functionality have also been
removed.
This is a partial revert of https://review.opendev.org/393886/
Change-Id: I9fc5868067e890321ad880cc134fb034b5401f47
Co-Authored-By: Rodolfo Alonso Hernandez <ralonsoh@redhat.com>
There is no real reason we should be using some of the
terms we do, they're outdated, and we're behind other
open-source projects in this respect. Let's switch to
using more inclusive terms in all possible places.
Change-Id: I99913107e803384b34cbd5ca588451b1cf64d594
"keepalived_state_change" monitor does not use eventlet but normal
Python threads. When "send_ip_addr_adv_notif" is called from inside
the monitor, the arping command is never sent because the eventlet
thread does not start. In order to be able to be called from this
process, this method should also have an alternative implementation
using "threading".
"TestMonitorDaemon.test_new_fip_sends_garp" is also modified to
actually test the GARP sent. The test was originally implemented with
only one interface in the monitored namespace.
"keepalived_state_change" sends a GARP when a new IP address is added
in a interface other than the monitored one. That's why this patch
creates a new interface and sets it as the monitor interface. When
a new IP address is added to the other interface, the monitor populates
it by sending a GARP through the modified interface [1].
[1] 8ee34655b8/neutron/agent/l3/keepalived_state_change.py (L90)
Change-Id: Ib69e21b4645cef71db07595019fac9af77fefaa1
Closes-Bug: #1870313
Use the "ip monitor" tool implemented with Pyroute2 library in
the neutron-keepalived-state-change monitor.
Change-Id: I932b62a8e0fa1a2f51bbde44134272f0b31b5c76
Related-Bug: #1680183
New IP command introduced by Ie3fe825d65408fc969c478767b411fe0156e9fbc
requires only privsep initialization. This patch removes the prisep
error FailedToDropPrivileges when executed under neutron-rootwrap.
Closes-Bug: #1823038
Change-Id: I6cde3c9dae7ffdccce49e88c3c79d1c379f291cf
Sometimes in case of HA routers it may happend that
keepalived will set status of router to MASTER before
neutron-keepalived-state-change daemon will spawn "ip monitor"
to monitor changes of IPs in router's namespace.
In such case neutron-keepalived-state-change process will never
notice that keepalived set router to be MASTER and L3 agent will
not be notified about that so router will not be configured properly.
To avoid such race condition neutron-keepalived-state-change will
now check if VIP address is already configured on ha interface
before it will spawn "ip monitor". If it is already configured
by keepalived, it will notify L3 agent that router is set to
MASTER.
Change-Id: Ie3fe825d65408fc969c478767b411fe0156e9fbc
Closes-Bug: #1818614
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
We don't maintain a long standing log file for the helper process, so
any log messages that we issue from inside it just vanish after router
cleanup.
This patch enforces syslog logging for the daemon, revealing any errors
and other messages that may be triggered by it.
Closes-Bug: #1635629
Change-Id: I8712f5f3a34fc9f403dfba06e7a64bd9eb408c4a
It turns out that Neutron uses python-requests on just one place
and for no good reason. Given that loading Requests amount for 80MB
of RAM in my system, maybe it's worth removing our dependency on
requests.
Change-Id: I905252a55bd174ff1aa8bdec42fe32432746846f
When HA routers are deleted, neutron-keepalived-state-change was
killed by l3/ha_router using SIGKILL and orphaning its child process
'ip -o monitor'.
This patch implements a way to kill it gracefully through SIGTERM, and
also sending a SIGKILL if keepalived-state-change didn't die after a
10 seconds timeout. The SIGTERM handler in keepalived-state-change
kills the ip monitor process using SIGKILL.
It kills ip_monitor PID using kill instead of through IPMonitor.stop()
because the monitor runs as root and keepalived-state-change doesn't
(it drops its privileges after launching the monitor). Doing so, would
lead to a "Permission denied". We can safely kill the monitor this way
since it was started with respawn_interval set to None which means that
it won't be automatically respawned.
Closes-Bug: #1632099
Change-Id: I7385172a007fdd252ea3a1c03c58064160d07e9e
We rely on keepalived to send gratuitous ARPs when floating IP is added.
Older versions of keepalived up to 1.2.20 (exclusive) contain bug [1] where
keepalived does not send GARP on receiving SIGHUP. Unfortunately, newer
versions containing the fix are not packaged yet for some distributions
like RHEL or CentOS or Ubuntu Xenial, so this patch adds a workaround for
such distributions until new packages are available.
The patch also sets net.ipv4.ip_nonlocal_bind kernel parameter to 0 for
Snat and HA router namespaces in order to avoid sending gratuitous ARPs
for IP addresses that are not bound to the interface anymore - possibly
because of failover or removal. Note that kernel < 3.19 contain a bug
where this knob is missing. In case it attempts to set the parameter and
it's missing on the system, it doesn't set the knob in root
namespace like it's done for fip namespaces, but only issues a warning
message.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1391553
Change-Id: Ieab53624dc34dc687a0e8eebd84778f7fc95dd77
Closes-bug: 1639315
Refactoring configuration options for agent/l3/keepalived_state_change
to be in neutron/conf/agent/l3. This would allow centralization of all
configuration options under neutron/conf and provide an easy way to
import.
Change-Id: I18e5f683690cffd6e0ad5e35f459b86c5b3332ac
Partial-Bug: #1563069
The option was deprecated a long time ago, and will be removed in one of
the next library releases, which will render neutron broken if we keep
using the option.
More details:
http://lists.openstack.org/pipermail/openstack-dev/2016-May/095166.html
Closes-Bug: #1586066
Change-Id: I884b4cc3ed04e4b5489e265c146666e04eb1bc27
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
Previously L3 HA generated a bash script and copied it to a per-router
configuration directory that was visible to that router's keepalived
instance. This patch changes the in-line generated Bash script to a
Python script that can be maintained in the repository.
The bash script was used as a keepalived notifier script, that was invoked
by keepalived whenever a state transition occured. These notifier scripts
may be invoked by keepalived out of order in case it transitions quickly
twice. For example, if the master failed and two slaves fight for the new
master role. One will transition to master, and the other will often
transition to master and then immidiately back to standby. In this case,
the transition scripts were often fired out of order, resulting in the
wrong state being reported.
The proposed approach is to get rid of the keepalived notifier scripts
entirely. Instead, monitor IP changes on the HA device. If the omnipresent
IP address was configured on the HA device, it means that we're looking
at a master instance. If it was deleted, the router transition to standby
or fault.
In order to keep the L3 agent CPU usage down, it will spawn a process
per HA router. That process will start the ip address monitor.
Whenever it gets an IP address change event, it will notify the L3 agent
via a unix domain socket.
Partially-Implements: blueprint report-ha-router-master
Change-Id: I2022bced330d5f108fbedd40548a901225d7ea1c
Closes-Bug: #1402010
Closes-Bug: #1367705
LIU Yulong