Implement the "kill" method (send a signal to a process) using the
Python native library "os".
In functional tests, "RootHelperProcess.kill" method should not fail if
the process does not exist.
Closes-Bug: #1843446
Closes-Bug: #1843418
Change-Id: Iee97a83779dd3e20eb3a223fb8557a94b8f15dc0
Replace rootwrap execution with privsep context execution.
This series of patches will progressively replace any
rootwrap call.
This patch migrates the "kill_process" method to privsep and
removes the unneeded rootwrap filters.
Change-Id: I48461be8b08cbc21c8af371f551b944343ba37bf
Story: #2007686
Task: #41558
Replace rootwrap execution with privsep context execution.
This series of patches will progressively replace any
rootwrap call.
This patch replaces some "IpNetnsCommand" command execution
methods.
Change-Id: Ic5fdf221a2a2cd0951539b0e040d2a941feee287
Story: #2007686
Task: #41558
Since [1], Neutron sets the name of some processes (Neutron agents).
The "ps" output is modified consequently according to the defined
string:
"<process name> (<process command>)"
"AsyncProcess" class should use the process name to parse the "ps"
output correctly.
Closes-Bug: #1902678
[1]https://review.opendev.org/#/c/735125/
Change-Id: If33c49c0f3e1e6696f5d2aa4008b287dc3f76c61
Neutron is python 3 only so these can be removed.
Another step in removing all of six usage from neutron.
Change-Id: Ica0913e689bb5b472053661b30f951477d3ec960
Now that we are python3 only, we should move to using the built
in version of mock that supports all of our testing needs and
remove the dependency on the "mock" package.
This patch moves all references to "import mock" to
"from unittest import mock". It also cleans up some new line
inconsistency.
Fixed an inconsistency in the OVSBridge.deferred() definition
as it needs to also have an *args argument.
Fixed an issue where an l3-agent test was mocking
functools.partial, causing a python3.8 failure.
Unit tests only, removing from tests/base.py affects
functional tests which need additional work.
Change-Id: I40e8a8410840c3774c72ae1a8054574445d66ece
Since it's no longer supported past Train, lets stop
running the tests.
Updated docs and made some pep8 code tweaks as well.
Change-Id: I1c171ab906a3b4c66558163ad26947ebf710a276
Even though we check for the existence of the process before, it may
still terminate before we get to read its command line, so catch the
error that can occur here.
Change-Id: I3e89aca8bedfd2912effe2490718223f7d03133e
Closes-Bug: 1844500
This patch switches the code over to use neutron-lib's test tools module
where appropriate rather than using neutron's.
This includes removing the following functions/classes from neutron and
using them from lib instead:
- get_random_EUI
- get_random_ip_network
- reset_random_seed
- OpenFixture
Change-Id: I0fbfcc7919f1b17b6bb0026fa9b98f157168255e
In patch [1] get_cmdline_from_pid function was modified to be able
to parse process' cmdline files with arguments separated by space
instead of '\0' char.
This patch adds one extra UT and small refactor which was pointed
in comments to [1] but which I was not able to change there.
[1] https://review.openstack.org/#/c/647605/
Change-Id: Ibd91d0472a686eca79a1126154d9cdf4587c1a19
Related-Bug: #1820870
According to proc man page process arguments in /proc/{pid}/cmdline
should be separated with '\0' char and that char was used in
neutron.agent.linux.utils.get_cmdline_from_pid function.
Recently in fullstack tests it was noticed that sometimes it may
happend that those arguments are separated with space char and this
caused failed test because async_process.AsyncProcess() was not able
to check that process is really active.
This patch adds attempt to split cmdline arguments with space in case
when split with '\0' returns only 1 element.
Change-Id: I35d4c0e2cf56fc3ff15cf307aaf11a8ad8489e1f
Closes-Bug: #1820870
Today the neutron common exceptions already live in neutron-lib and are
shimmed from neutron. This patch removes the neutron.common.exceptions
module and changes neutron's imports over to use their respective
neutron-lib exception module instead.
NeutronLibImpact
Change-Id: I9704f20eb21da85d2cf024d83338b3d94593671e
Currently, the neutron-openvswitch-agent does not start on Windows
due to Linux specific imports. This patch addresses this issue.
Also, we're wrapping the object returned by subprocess.Popen using
tpool.Proxy in order to prevent IO operations on the stream
handles from blocking other threads. Currently, the ovs db monitor
blocks the whole process.
Closes-Bug: #1775382
Co-Authored-By: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Change-Id: I8bbc9d1f8332e5644a6071f599a7c6a66bef7928
It was recently decided to uncap eventlet:
http://lists.openstack.org/pipermail/openstack-dev/2018-April/129096.html
So eventlet is now capped at 0.20 not by global requirements,
it is capped in upper-constraints, because currently not every
openstack project is able to work with a newer eventlet version,
mostly because of the caps in projects requirements.txt.
According to global-requirements, last allowed version of
eventlet is 0.22.1:
https://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt
In an effort to support both eventlet<0.22 and eventlet>=0.22,
change the code to try and determine the correct number of
arguments to use in the call to initialize the parent class.
Change-Id: Ibe3dc8af6cf9f8bb4f8eababb7f4276e4db3f1f9
Closes-bug: #1777640
If the rootwrap daemon fails to execute a command, it
generates a cryptic message:
Unserializable message: ('#ERROR', ValueError('I/O operation on closed file',))
We should at least log the command that we were trying
to run, which will help users figure out why it failed.
Change-Id: I2c94e5a226630432028351f8287868f4fe5d2fa1
Closes-bug: #1677742
We have made os-xenapi repository to deal with XenServer Dom0
specific functions, this patch is to change neutron to use
os-xenapi when XenServer is hypervisor and move the building
RPM scripts into os-xenapi repo
Depends-On: I8a31c81d9475387fe4ed7030b70b26098e588771
Change-Id: Ia958c366189386b1b5abbadbb4d74950aaa23bb2
utils.kill_process() parsed the error string after the kill command
which can lead to internationalization issues. We shouldn't relay upon
different translations so this follow-up patch removes this dependency
by checking if the process is still running after the kill when a
ProcessExecutionError exception occurs.
Before, this was achieved by comparing against "No such process" string
Change-Id: I22bd63992d1029f99fea401f07167383f8ff7dd0
Removing the deprecated method get_interface_mac and from
neutron/agent/linux/utils.py and the associated test in
neutron/tests/unit/agent/linux/test_utils.py. This is scheduled
for removal in Pike.
This was deprecated in I1695d7e46efe5245eb581bd40d5420250a3bad89.
Change-Id: I6b84563c2631a3e47826320f03fa1fdfe44cf2a9
For Neutron's compute agent in a XenServer's compute node, the commands
actually need run in Dom0. Currently XenServer only supports rootwrap
for that purpose by invoking a script which invokes XenAPI to execute
commands in dom0. There are much performance overhead due to it requires
parsing on the script and the configuration file every time running
commands.
This change is to support daemon mode with which each agent service will
call XenAPI directly to execute commands in dom0. And it will keep the
single XenAPI session.
DocImpact: Need update the following configuration.
file: /etc/neutron/plugins/ml2/openvswitch_agent.ini
[agent]
root_helper_daemon = xenapi_root_helper
[xenapi]
connection_url = http://169.254.0.1
connection_username = root
connection_password = xenroot
Closes-Bug: #1585510
Change-Id: I684034359fe0571bc92dbcf342a9821553b1da35
This patch will kill processes that are listening on any port/UNIX
socket within the namespace to be cleaned up. To kill them it will
issue a SIGTERM to them (or to their parents if they were forked) and,
if they don't die after a few seconds, a SIGKILL to them and all their
children.
This is intended for those cases when there's no specific cleanup and
serves as a fallback method.
Change-Id: I4195f633ef4a1788496d1293846f19eef89416aa
Partial-Bug: #1403455
Currently, execute() may raise an exception that contains a *translated*
string that starts with 'Exit code: %(returncode)d...' if the returncode
of a process was not 0. find_child_pids() will then check if the
raised exception contains 'Exit code: 1' (to check if the returncode is
1), but in non-English locales this will fail as the 2 strings are not
encoded the same.
This patch adds a new ProcessExecutionError (which inherits from
RuntimeError, so as to not change all the code that currently depends on
execute() returning RuntimeError) which now accepts a returncode. This
can be changed explicitly without depending on the error message.
Later patches can move ProcessExecutionError to neutron-lib, if this is
needed - this patch intends to write the smallest piece of code that can
be backported.
Closes-Bug: #1638273
Change-Id: I85d3bec13e852918eb13e73c1367c70e1f4d34b1
Currently max number of client connections(i.e greenlets spawned at
a time) opened at any time by the WSGI server is set to 100 with
wsgi_default_pool_size[1].
This configuration may be fine for neutron api server. But with
wsgi_default_pool_size(=100) requests, state change server
is creating heavy cpu load on agent.
So this server(which run on agents) need lesser value i.e
can be configured to half the number of cpu on agent
We use "ha_keepalived_state_change_server_threads" config option
to configure number of threads in state change server instead of
wsgi_default_pool_size.
[1] https://review.openstack.org/#/c/278007/
DocImpact: Add new config option -
ha_keepalived_state_change_server_threads, to configure number
of threads in state change server.
Closes-Bug: #1581580
Change-Id: I822ea3844792a7731fd24419b7e90e5aef141993
Reactor code:
* Reuse oslo_utils.encodeutils.to_utf8() instead of existing
isinstance(text, six.text_type) test
* Replace jsonutils.dumps(obj).encode('utf-8') with
jsonutils.dump_as_bytes(obj).
* Other minor bytes/Unicode changes
Change-Id: I03b8eff0fd70ab65ac66d6f3221e8ced0a56db17
get_root_helper_child_pid recursively finds the child of pid,
until it can no longer find a child. However, the intention is
not to find the deepest child, but to strip away root helpers.
For example 'sudo neutron-rootwrap x' is supposed to find the
pid of x. However, in cases 'x' spawned quick lived children of
its own (For example: ip / brctl / ovs invocations),
get_root_helper_child_pid returned those pids if called in
the wrong time.
Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
Closes-Bug: #1558819
Commit I26b0a4d6105420a2c242b81a4cd58e0adef4cbec marked method
replace_file as redundant. Functionality was moved to
neutron.common.utils:replace_file
Related-Bug: #1504477
Change-Id: I77f907bee20bf921d4127502c1ce8156425e158a
Process output is supposed to be represented with lines, so we should
put Python strings in the queue (not bytes). Just in case, we do it only
for Python 3 environment.
To fix that, we reuse code from utils.execute() linux/windows
implementations.
This fixes the TestAsyncProcess.test_async_process_respawns functional
test for Python 3 environment.
Related-Bug: #1515118
Change-Id: I9efec2290003add44909aab33a0026372a580016
The commit 048316e981 introduces the
pattern:
if isinstance(line, bytes):
try:
line = line.decode(encoding='utf-8')
except UnicodeError:
pass
# concat line with a string
which is not working in PY3K if an UnicodeError is raised because line
is (silently) not decoded and concatened to a string.
This change ensures to return a text object or to raise an error.
Closes-Bug: #1503415
Blueprint: neutron-python3
Change-Id: I16b8013f33aa3efad65be8040d3210120e047bbd
The replace_file() utility function currently sets the mode of all files
it creates to 0o644. This is not appropriate for all files. This patch
adds an optional "file_mode" argument to the function.
Change-Id: I9744abde10b95fadef6e74c55332d041e5372071
Partial-Bug: 1488320
In Python 3, input and output for Popen.communicate() is bytes type.
Therefore, encode input data and decode return data for Popen.communicate().
Change-Id: I70f009e3366f0eeda5790652ea14f3627b934664
Blueprint: neutron-python3
Closes-Bug: #1479159
There is nothing Linux or agent specific in the function. I need to use
it outside agent code in one of depending patches, hence moving it into
better location while leaving the previous symbol in place, with
deprecation warning, for backwards compatibility.
Change-Id: I252356a72f3c742e57c1b6127275030f0994a221
I224be69168ede8a496a5f7d59b04b722f4de7192 added an EEXIST
check, so no need to check if the directory is already
there, just try and create it.
Change-Id: Iba51fc8263bf59326489319d0dd3f69af00a8eeb
In rare cases, concurrent workers may attempt to ensure a directory
exists. One may successfully create the directory while the other
gets an oserror that it already exists. This patch detects the
problem and returns successfully in both cases.
Change-Id: I224be69168ede8a496a5f7d59b04b722f4de7192
Change eba4c2941e introduced these tests. However they are not that useful as they
simply mimick the code, without really ensuring that the behavior is expected, so
they provide negative value ([1]), plus, they fail randomly.
This patch removes them in favor of a more useful functional check.
[1] http://googletesting.blogspot.com/2015/01/testing-on-toilet-change-detector-tests.html
Closes-bug: #1441347
Change-Id: I8a321995295deef7f6d30be303486be491e2771f
Currently metadata proxy cannot run with nobody user/group as metadata
proxy requires to connect to metadata_proxy_socket when queried.
This change allows to run metadata proxy with nobody user/group by
allowing to choose the metadata_proxy_socket mode with the new option
metadata_proxy_socket_mode (4 choices) in order to adapt socket
permissions to metadata proxy user/group.
This change refactors also where options are defined to enable
metadata_proxy_user/group options in the metadata agent.
In practice:
* if metadata_proxy_user is agent effective user or root, then:
* metadata proxy is allowed to use rootwrap (unsecure)
* set metadata_proxy_socket_mode = user (0o644)
* else if metadata_proxy_group is agent effective group, then:
* metadata proxy is not allowed to use rootwrap (secure)
* set metadata_proxy_socket_mode = group (0o664)
* set metadata_proxy_log_watch = false
* else:
* metadata proxy has lowest permissions (securest) but metadata proxy
socket can be opened by everyone
* set metadata_proxy_socket_mode = all (0o666)
* set metadata_proxy_log_watch = false
An alternative is to set metadata_proxy_socket_mode = deduce, in such
case metadata agent uses previous rules to choose the correct mode.
DocImpact
Closes-Bug: #1427228
Change-Id: I235a0cc4f0cbd55ae4ec1570daf2ebbb6a72441d
Currently metadata proxy cannot run with nobody user/group as
metadata proxy (as other services) uses WatchedFileHandler handler to
log to file which does not support permissions drop (the process must
be able to r/w after permissions drop to "watch" the file).
This change allows to enable/disable log watch in metadata proxies with
the new option metadata_proxy_log_watch. It should be disabled when
metadata_proxy_user/group is not allowed to read/write metadata proxy
log files. Option default value is deduced from metadata_proxy_user:
* True if metadata_proxy_user is agent effective user id/name,
* False otherwise.
When log watch is disabled and logrotate is enabled on metadata proxy
logging files, 'copytruncate' logrotate option must be used otherwise
metadata proxy logs will be lost after the first log rotation.
DocImpact
Change-Id: I40a7bd82a2c60d9198312fdb52e3010c60db3511
Partial-Bug: #1427228
Rodolfo Alonso