The backdoor command fo() uses isinstance() to check if an object is an
instance of a class. This only works with objects that have a __class__
attribute, else an AttributeError is raised by isinstance(). This is
seldomly the case, though if there is one such object fo() will cease to
work. Therefore we need to protect us against this case by checking for
a __class__ attribute before calling isinstance().
An example for an object without __class__ would be
functools._lru_list_elem.
Change-Id: Ia4c5cbdc249535d36f6e71f7b2a7359bc6fdf219
Closes-Bug: #1946072
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: Icc2277b72f6f8f7812be22c43bbc281334aa2373
Closes-Bug: #1863021
When multiple processes are spawned with the same configuration
each process has the same backdoor_socket path configured and
only the first process able to bind to the socket can later be
accessed via the backdoor. To give each process a unique socket path
we now expose the PID of the process as a format string argument,
which can then be used like this:
backdoor_socket = /var/lib/neutron/backdoor-{pid}
Change-Id: I3f86f4867eb0cd5010abadf68620aa3450d3e64d
This patch enables profiling (capturing function call trace like
cProfile [1]) worker processes on the fly while service is running.
User requests the oslo service process to start profiling by writing
"prof()" command to backdoor socket, once the service (like
neutron-server) finishes expected processing (example finishing API
call), user again writes "prof()" command with file name as argument
to dump the function calltrace stats. Stats file (in pstat format
with user provided filename by adding .prof) will be generated in
temp directory.
For example, to profile neutron server process,
1) echo "prof()" | nc localhost 8002
2) Issue neutron command (or run rally scenarios tests)
neutron net-create n1
neutron port-create --name p1 n1
neutron port-delete p1
neutron net-delete n1
3) echo "prof('neutron')" | nc localhost 8002
where 8002 is the port which we set like below in neutron.conf
backdoor_port=8002
We can later print the stats from the trace file like below
stats = pstats.Stats('/tmp/neutron.prof')
stats.print_stats()
The trace file will look like in (for above neutron API calls) [2].
We use Yappi with context set to greenlet [3] to profile greenlets.
We can't use GreenletProfiler [4], which does the same [5]
1) as it is no more maintained
2) Also compiling yappi source inside GreenletProfiler is failing for
python3.
[1] https://docs.python.org/2/library/profile.html
[2] https://gist.github.com/venkataanil/64d5e672bf0206dc151e73fc1058a983
[3] https://bitbucket.org/sumerc/yappi/pull-requests/3
[4] https://pypi.org/project/GreenletProfiler/
[5] https://emptysqua.re/blog/greenletprofiler/
Depends-On: Ibea0cdb732923f1b53d5cb6aeeb4041fb5973494
Change-Id: Id2418093494f1e233a653f6c73bd6894e4a40184
Oslo.service is binding to same port when we provide a port range
as eventlet is internally setting SO_REUSEPORT flag (starting from
eventlet version v0.20). And there is a flag (reuse_port)
introduced in v0.22 to give control to user to avoid SO_REUSEPORT.
In this patch, first we try passing reuse_port=False, if this fails
then directly open socket and listen instead of eventlist.listen.
Closes-Bug: #1810280
Change-Id: Idc842acc7e430199c76fe12785b0bf0e7a58e121
Local files can be made accessible to certain users vs random
ports which can be accessible to anyone on a machine so allow
using unix files as a way to start the eventlet backdoor (so that
user and group permissions common on unix are not lost).
To use this new type of files `socat` is needed (or other way
to interact with telnet over a unix domain socket).
For example (with the path at /tmp/my_special_socket):
socat - UNIX-CONNECT:/tmp/my_special_socket
Depends-On: Ia2385879e09991102f8f305ec41dbb651a4374de
Change-Id: I7f25913168ebe5854f360db3d6310b72a56b2b4d
When things start behaving badly having access to a more
capable frame dumping function can be pretty handy to gain
more insight into what the greenthreads are doing (and what
the functions and those functions local variables are that
they are currently running with).
This adds such a capability to the pgt() alias function that
the backdoor exposes, the default usage shows the simpler and
less useful print_stack() output but the newer and less simple
output can be enabled by calling pgt(False).
Doing so will produce output like:
https://gist.github.com/harlowja/b91b512dcf1b6db592bb
Change-Id: Ifc77b908699b911f60f039ac54f98391a7e53732
In order to determine what to do about this module it
is quite useful to provide for a way to run it as its own
main program (and interact with it from another telnet
session).
This adds a simple `main` function and uses it.
This can then be used like:
$ python oslo_service/eventlet_backdoor.py --backdoor_port 8000
Then from another terminal:
$ telnet localhost 8000
Change-Id: I8c2b93dfad328929e1a106c5883903fe0bfb188a
Change the API of oslo.service to accept conf object and
remove usage of global CONF object throughout the library.
List of functions/classes that accept conf as a parameter:
1. eventlet_backdoor.py
* initialize_if_enabled
2. periodic_task.py
* PeriodicTasks
3. service.py
* launch
* ServiceLauncher
* ProcessLauncher
4. sslutils.py
* is_enabled
* wrap
Fixed the unit tests.
APIImpact
bp graduate-oslo-service
Change-Id: Icbd5f9e386b0eee2f7c60a65c076f53ba5f0f637
It is not always desirable to log full set of conf at service
start. This change adds new option "log_options" that allows to
enable or disable logging values of all registered options.
Also moved list_opts function for service submodule to service.py
as service and eventlet_backdoor opts have to be grouped together
and changed the entrypoint accordingly.
Change-Id: I2a97ebf736fd361e6f1d05796d5077bc9627ff85
Closes-Bug: #1461250
Sebastian Lohff