Due to the unordered nature of sets, it is possible for signal
handlers like _gracefull_shutdown() to run first and clear the
_signal_handlers before other registered handlers have run.
This patch converts _signal_handlers from a set to a list and
then runs the handlers LIFO similar to how atexit functions are
run. Given that the handlers added in service.py are added after
clearing the signal_handlers, they are guaranteed to be the first
in the list and the last run.
Change-Id: Ief3ab2f56fa3d805296709e12e5f7a71560f7b6e
Closes-Bug: #2057809
Related-Bug: #2056366
Signed-off-by: Terry Wilson <twilson@redhat.com>
When the parent process used eventlet tpool to run code in native
threads then anything the child tries to run in a native thread will
hang forever.
The reason for that is that the parent has initialize the pool of
threads and use a flag to mark that it has been initialized, and when
the child is forked then it will have the flag saying that it has been
initialized and expects the threads to be running, but they are not. So
there is no thread to pick up the job when a greenlet queues the job, so
the greenthread waits forever to get the result of the operation.
This patch tells eventlet's tpool to clean things up on the child just
after forking, that way if the child uses native threads tpool will
spawn all the threads again.
Closes-Bug: #1983949
Change-Id: If2421427c48faa976d6c6ee9bafe4d563288037b
The backdoor command fo() uses isinstance() to check if an object is an
instance of a class. This only works with objects that have a __class__
attribute, else an AttributeError is raised by isinstance(). This is
seldomly the case, though if there is one such object fo() will cease to
work. Therefore we need to protect us against this case by checking for
a __class__ attribute before calling isinstance().
An example for an object without __class__ would be
functools._lru_list_elem.
Change-Id: Ia4c5cbdc249535d36f6e71f7b2a7359bc6fdf219
Closes-Bug: #1946072
The backoff looping call logging was previously making
a decision on if timeout had or was going to be exceeded
by including idle time and it's internal time, however
this is misleading as the overall operation may take
30 seconds externally, a user initiated timeout of 30
seconds is requested, and the error might say something
like 18 seconds has passed, when that is incorrect.
The logic is actualy correct, the logging was just
misleading.
Fixes the exception message to use the total time.
Change-Id: Ie9ef5a53abb24f6ab7de0ad57a672c0a8d7ff0ee
Eventlet uses monotonic clocks by default since 0.21, hence no need
to patch that anymore.
Closes-Bug: #1906262
Co-Authored-By: Dimitri John Ledkov <xnox@ubuntu.com>
Co-Authored-By: Michael Hudson-Doyle <michael.hudson@ubuntu.com>
Change-Id: I72106068d99cc174412aa3026082ec63f1d895fe
Introduced changes:
- pre-commit config and rules
- Add pre-commit to pep8 gate, Flake8 is covered in the pre-commit hooks.
- Applying fixes for pre-commit compliance in all code.
Also commit hash will be used instead of version tags in pre-commit to
prevend arbitrary code from running in developer's machines.
pre-commit will be used to:
- trailing whitespace;
- Replaces or checks mixed line ending (mixed-line-ending);
- Forbid files which have a UTF-8 byte-order marker
(check-byte-order-marker);
- Checks that non-binary executables have a proper
shebang (check-executables-have-shebangs);
- Check for files that contain merge conflict strings
(check-merge-conflict);
- Check for debugger imports and py37+ breakpoint()
calls in python source (debug-statements);
- Attempts to load all yaml files to verify syntax (check-yaml);
- Run flake8 checks (flake8) (local)
For further details about tests please refer to:
https://github.com/pre-commit/pre-commit-hooks
Change-Id: Ib18f38dbec90c62e870307bf22a8b4f193237bce
Signed-off-by: Moisés Guimarães de Medeiros <moguimar@redhat.com>
The LOG instance is not being used and there is no need to import
"oslo_log". This will also prevent the circular import detected
in the related bug.
Change-Id: Id560ba80472b9efbcfbc289a18d4df312961fdb1
Related-Bug: #1888213
Previously some tests were ignored under python 3 environment, this was
due to some design changes introduced by python 3.7 [1] in the SSL
module of the stdlib. These changes reactivate some of them (some other
are still skipped and needs further works).
Indeed, when we try to use requests with SSL in a monkey patched
environment we faced the following issue:
```
TypeError: wrap_socket() got an unexpected keyword argument '_context'
```
This is due to the fact that we are in a monkey patched environment
where `requests` is monkey patched too.
We don't need `request` for our needs. Indeed we can easily send
http requests through low level socket. Our main goal is to test
our wsgi server and not to test the `requests` library, and `requests`
was just used to make the code more simpler.
In our case we can implement a code dedicated to send request to our green
server, unlock our tests and move away from this bug/side effect.
Also this reactivated test will check WSGI server with and without SSL,
so these changes add changes that allow us to submit a request without
wrapping the socket with SSL.
These changes move away from `requests` which is badly monkey patched by
eventlet [1]. Now we use monkey patched socket and ssl to
request the green server which is executed in background. Low level
(monkey patched) modules could help us to skirt layers that are possibly
badly monkey patched on higher level modules (urllib, requests, etc...).
[1] https://github.com/eventlet/eventlet/issues/526#issuecomment-482694279
[2] https://github.com/eventlet/eventlet/issues/526
Change-Id: I3a018d507d102266c1e2fc9b6732a9c09fa2bb49
Closes-Bug: #1482633
Previously these tests was ignored surely because the WSGI tests
with SSL were broken due to some design changes introduced by python 3.7 [1]
in the SSL module of the stdlib.
However, unlike the other WSGI/SSL tests these changes don't use
the `requests` module or high level modules which are broken by
the monkey patching of the stdlib.
I suppose previous developers simply turned off all of tests related to
WSGI/SSL without much more details than "SSL tests with python 3 and eventlet
is broken" but some of them don't need much more work than just reactivate
them to be ran successfully.
This test seems to work without change, and it run successfully locally.
Let reactivate this test.
Change-Id: Ie0257af10a9439de8ec124a698ca75778bdf90b5
Partial-Bug: #1482633
Previously some tests were ignored under python 3 environment, this was
due to some design changes introduced by python 3.7 [1] in the SSL
module of the stdlib. These changes reactivate some of them (some other
are still skipped and needs further works).
Indeed, when we try to use requests with SSL in a monkey patched
environment we faced the following issue:
```
TypeError: wrap_socket() got an unexpected keyword argument '_context'
```
This is due to the fact that we are in a monkey patched environment
where `requests` is monkey patched too.
We don't need `request` for our needs. Indeed we can easily send
http requests through low level socket. Our main goal is to test
our wsgi server and not to test the `requests` library, and `requests`
was just used to make the code more simpler.
In our case we can implement a code dedicated to send request to our green
server, unlock our tests and move away from this bug/side effect.
These changes move away from `requests` which is badly monkey patched by
eventlet [1]. Now we use monkey patched socket and ssl to
request the green server which is executed in background. Low level
(monkey patched) modules could help us to skirt layers that are possibly
badly monkey patched on higher level modules (urllib, requests, etc...).
[1] https://github.com/eventlet/eventlet/issues/526#issuecomment-482694279
[2] https://github.com/eventlet/eventlet/issues/526
Change-Id: Iacdde51d2d923bafa3263fa3dc12de8d501d471a
Partial-Bug: #1482633
Previously some tests were ignored under python 3 environment, this was
due to some design changes introduced by python 3.7 [1] in the SSL
module of the stdlib. These changes reactivate some of them (some other
are still skipped and needs further works).
Indeed, when we try to use requests with SSL in a monkey patched
environment we faced the following issue:
```
TypeError: wrap_socket() got an unexpected keyword argument '_context'
```
This is due to the fact that we are in a monkey patched environment
where `requests` is monkey patched too.
We don't need `request` for our needs. Indeed we can easily send
http requests through low level socket. Our main goal is to test
our wsgi server and not to test the `requests` library, and `requests`
was just used to make the code more simpler.
In our case we can implement a code dedicated to send request to our green
server, unlock our tests and move away from this bug/side effect.
These changes move away from `requests` which is badly monkey patched by
eventlet [1]. Now we use monkey patched socket and ssl to
request the green server which is executed in background. Low level
(monkey patched) modules could help us to skirt layers that are possibly
badly monkey patched on higher level modules (urllib, requests, etc...).
[1] https://github.com/eventlet/eventlet/issues/526#issuecomment-482694279
[2] https://github.com/eventlet/eventlet/issues/526
Change-Id: Id44ad12a1cf3fd7090a67bb6e8e42bfdc47502cf
Partial-Bug: #1482633
Recent changes [1] introduced an eventlet fix to monkey patch original
current_thread _active.
The goal was to monkey patch the original current_thread to use the
up-to-date _active global variable. This solution is based on that
documented at: eventlet/eventlet#592
I think we need this patch on unit test too to ensure a consistent
behavior, so these changes doing that.
[1] https://review.opendev.org/#/c/725359/
Change-Id: I7b6cca86e44261bf2f953be74e9738ac09507649
The monotonic package was needed for monotonic time operations when
running under Python runtimes older than 3.3. Since we now only support
versions higher than this, this third party package requirement can now
be removed.
Change-Id: Ie99411882822a5db47462fcb2748c6e978587b8c
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: Icc2277b72f6f8f7812be22c43bbc281334aa2373
Closes-Bug: #1863021
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: Ic18f03681c14131217e3025e2cb0cca46ada84ee
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Resolves the following warnings seen in the likes of nova:
DeprecationWarning: Please provide `is_available()` function in your
custom Eventlet hub <module 'oslo_service' from
'../oslo_service/__init__.py'>."
It must return bool: whether hub supports current platform. See
eventlet/hubs/{epoll,kqueue} for example.
Since we're simply taking the default hub provided by eventlet and
overriding a single attribute, we will always have support so set the
attribute.
Change-Id: I219cf652526802b8212fe8a8f6c499e9aa24a3cf
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
to make tests pass after 2023
Background:
As part of my work on reproducible builds for openSUSE, I check
that software still gives identical build results in the future.
The usual offset is +15 years, because that is how long I expect
some software will be used in some places.
This showed up failing tests in our package build.
See https://reproducible-builds.org/ for why this matters.
Note: I tested that i586 builds still pass tests on 2037-12-01
so this change introduces no year2038 problem.
Change-Id: I0110914432c4b557dcabcac0fad1ba54bcd8be85
It seems that the code for handling SIGHUP currently calls stop()
on the service, then calls reset(), then calls start() on it again.
This is effectively a full service restart, which breaks the whole
point behind using SIGHUP for hot and quick reloads. It also breaks
our downstream projects in a few ways where they lose RPC on reload
due to the fact that they don't expect to have stop() called on a
reset().
This patch removes the stop and start when the restart_method is
set to 'mutate' because in that case we should just be signaling
the service to check for changes in its mutable config options.
It also changes the signal sent to children in that case to
SIGHUP, since SIGTERM will cause unnecessary restarts of child
processes.
The previous behavior is maintained for the 'reload' restart_method
since that does a complete reload of the service config, which is
not safe to do without restarting the service completely.
Change-Id: I86a34c22d41d87a9cce2d4ac6d95562d05823ecf
Closes-Bug: #1794708
Co-Authored-By: Ben Nemec <bnemec@redhat.com>
When multiple processes are spawned with the same configuration
each process has the same backdoor_socket path configured and
only the first process able to bind to the socket can later be
accessed via the backdoor. To give each process a unique socket path
we now expose the PID of the process as a format string argument,
which can then be used like this:
backdoor_socket = /var/lib/neutron/backdoor-{pid}
Change-Id: I3f86f4867eb0cd5010abadf68620aa3450d3e64d
If user set the type of worker to str or others, there will
be no error here, but a lot of processes will be created,
which may cause the system to crash.
I recommend adding a type check here.
Change-Id: I3ad0f7ec59f29a3106d23b057327c3dfef19a98f
This patch enables profiling (capturing function call trace like
cProfile [1]) worker processes on the fly while service is running.
User requests the oslo service process to start profiling by writing
"prof()" command to backdoor socket, once the service (like
neutron-server) finishes expected processing (example finishing API
call), user again writes "prof()" command with file name as argument
to dump the function calltrace stats. Stats file (in pstat format
with user provided filename by adding .prof) will be generated in
temp directory.
For example, to profile neutron server process,
1) echo "prof()" | nc localhost 8002
2) Issue neutron command (or run rally scenarios tests)
neutron net-create n1
neutron port-create --name p1 n1
neutron port-delete p1
neutron net-delete n1
3) echo "prof('neutron')" | nc localhost 8002
where 8002 is the port which we set like below in neutron.conf
backdoor_port=8002
We can later print the stats from the trace file like below
stats = pstats.Stats('/tmp/neutron.prof')
stats.print_stats()
The trace file will look like in (for above neutron API calls) [2].
We use Yappi with context set to greenlet [3] to profile greenlets.
We can't use GreenletProfiler [4], which does the same [5]
1) as it is no more maintained
2) Also compiling yappi source inside GreenletProfiler is failing for
python3.
[1] https://docs.python.org/2/library/profile.html
[2] https://gist.github.com/venkataanil/64d5e672bf0206dc151e73fc1058a983
[3] https://bitbucket.org/sumerc/yappi/pull-requests/3
[4] https://pypi.org/project/GreenletProfiler/
[5] https://emptysqua.re/blog/greenletprofiler/
Depends-On: Ibea0cdb732923f1b53d5cb6aeeb4041fb5973494
Change-Id: Id2418093494f1e233a653f6c73bd6894e4a40184
Instead of having a copy-pasted version in this project, let's just
use the original directly. It is added to the public API of
oslo.utils in the dependency.
Depends-On: https://review.openstack.org/614806
Change-Id: If0dfac2505d097c117ef94c99399b1614f1e1f8f
Oslo.service is binding to same port when we provide a port range
as eventlet is internally setting SO_REUSEPORT flag (starting from
eventlet version v0.20). And there is a flag (reuse_port)
introduced in v0.22 to give control to user to avoid SO_REUSEPORT.
In this patch, first we try passing reuse_port=False, if this fails
then directly open socket and listen instead of eventlist.listen.
Closes-Bug: #1810280
Change-Id: Idc842acc7e430199c76fe12785b0bf0e7a58e121
This API is awkward, inefficient, incoherent, and unintuitive. The bug
fix for which it was originally added was abandoned in favour of a
different approach, so it was never used. It appears that no consumers
are currently calling it. It would be best if none started.
Add a deprecation warning to discourage its use and allow us to remove
it altogether at some point in the future.
Change-Id: I9559c7051024019fac957385faced645920b815c
There were no docstrings for this API, which as a bonus is extremely
subtle and poorly-designed. Document it so that users at least have a
fighting chance of using it to do the things they intended.
Change-Id: Id6677a7e5aaf8ec9ddeb597c6d5e8f97806e2248
The intention of this test was to wait 5s after sending SIGHUP to a
child process to make sure that it doesn't exit. However, due to a logic
error, it just stopped checking and declared success immediately. Fix
the logic so that we have a better chance of seeing if SIGHUP
incorrectly kills the process.
Change-Id: I1f320a8dfdd7a922b461d070491ad53e6cd2b20d
Related-Bug: #1803731
The patch 2ee3894f49 broke the original
fix cad75e4e13 that ensured eventlet could
be interrupted while sleeping after PEP475 was implemented in Python
3.5. Eventlet monkey-patches the signal module with its own version, so
we have to look up the original module to determine whether the
underlying OS actually supports the poll() function.
Change-Id: Ia712c9a83d8081bf0b5e6fe36f169f9028aae3dc
Closes-Bug: #1803731
Related-Bug: #1788022
Related-Bug: #1705047