Commit Graph

96 Commits

Author SHA1 Message Date
Stephen Finucane d17f58d1bf trivial: Remove unnecessary 'coding' lines
All files use UTF-8 by default in Python 3.

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I7acf3a0ad369371aa4075962b3656467a6a37c58
2022-04-06 14:14:45 +01:00
Stephen Finucane 57678fc04c Remove six
We don't need this dependency anymore

Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I93d0c05b1f63c86040b52cdaea99ffb42bad3f07
2022-04-06 14:11:43 +01:00
Sean McGinnis b29f99d67d
Remove monotonic usage
The monotonic package was needed for monotonic time operations when
running under Python runtimes older than 3.3. Since we now only support
versions higher than this, this third party package requirement can now
be removed.

Change-Id: I0e5bf6090ed85db2659b50096122a9abda9dde6a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-05-04 17:17:01 -05:00
Sean McGinnis cd07ab5eaf
Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: I2b8d8d6cf4ac29dc58cd8fe418d613445d65c3be
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-04-18 11:52:37 -05:00
Monty Taylor 79f240b79f Make PrettyTable optional
PrettyTable as a depend can have a tendency to conflict with other
things that want to use it, which is a shame for a general purpose
library like futurist. Make it optional to reduce the transitive
dependency burden for consumers.

Change-Id: Ie976f7f4e544f273fccf0cf579b82b659cbd48d1
2020-03-03 15:51:32 -06:00
Hervé Beraud d3cb905629 Fix Calling waiters.wait_for_any() blocks if future has called Condition.wait()
condition.wait() can block forever even if Futures have completed

A similar issue was already fixed on cpython few years ago:
- https://bugs.python.org/issue20319
- https://hg.python.org/cpython/rev/0bcf23a52d55
- 2b754f49a5

Change-Id: I1fbad12a0d555e199fdf4073db9408a296e7fd8a
Related-Bug: #1848457
2019-10-23 11:11:27 +02:00
Monty Taylor 658a29412b
Only install monotonic on python2
monotonic is a backport of time.monotonic from python3. Only install it
for python2.

Depends-On: https://review.openstack.org/615441
Change-Id: Id27cd748e883d54dd93dac2e6bd8caee6728f7e1
2018-11-04 08:42:59 -06:00
melissaml a8bcc17b10 Remove the duplicated word
Change-Id: Ib5fa0d39c0c19e168937b785f97e3c88c95d49cd
2018-08-22 12:07:52 +08:00
Chenjun Shen 8fff2242ff Improve get optimal count of max_worker for pool
In the upstream of futures library, the DEFAULT value max_worker
for ThreadPoolExecutor and ProcessPoolExecutor has been changed to:

ThreadPoolExecutor

max_worker = cpu_count() * 5

Reference:
https://github.com/agronholm/pythonfutures/blob/master/concurrent/futures/thread.py#L109

ProcessPoolExecutor

max_worker = cpu_count()

Reference:
https://github.com/agronholm/pythonfutures/blob/master/concurrent/futures/process.py#L278

So we should do it as well.

Change-Id: Id68e38a639c41426f125180ad335fb1fb97cba59
Closes-bug: #1739034
2017-12-21 16:01:17 +01:00
Jenkins fba366a839 Merge "Expose underlying timeout error" 2017-04-14 00:56:44 +00:00
Joshua Harlow dc17c8824e Expose underlying timeout error
This makes it easier to use the other various
functions in futures or executors that will raise
this when they timeout.

Change-Id: If18d55fc20b2cb6a919c9a1a98a818f4068bd3ec
2017-03-31 22:56:26 -07:00
Yuriy Zveryanskyy b8759972cb Do not check spacing if periodic disabled
This change allows simple enable/disable periodic and set spacing
via one configuration option, like
@periodics.periodic(spacing=CONF.task_interval,
                    enabled=CONF.task_interval > 0)

Change-Id: I1f00242b49c7646421390c0ff26777aba6d5b6d9
Closes-Bug: #1654035
2017-03-23 14:36:25 +02:00
Joshua Harlow e05b121e6a Introducing NeverAgain functionality for periodics
Allow functions to raise a never again exception to remove
themselves from running

It also introduces a new param named ``auto_stop_when_empty``
to ``start()`` method. When the provided periodic functions have
all exited and this param is set to false then the thread
responsible for executing those methods will just spin/idle
waiting for a new periodic function to be added; switching it to
true will make this idling not happen (and instead when no more
periodic work exists then the calling thread will just return).

Co-Authored-By: Debayan Ray <debayan.ray@gmail.com>

Change-Id: If7c5e1a407fb72af8e0f7f3a1017a123590cffb2
2017-01-23 18:36:45 -08:00
Dmitriy Ukhlov ee42ab588a Add re-raising SystemExit exception
SystemExit exception is usually raised by signal handlers
and then caught in main thread to handle signal. Signal
handler executes in main threads but if we use eventlet
we have ony one thread and signal handler can interrupt
any greenlet. Therefore we need to re-raise this exception
to catch it in main greenlet at last.

Also it is necessary to keep executor working and continue
processing scheduled tasks because signal handling logic may
wait for some job results

Change-Id: Ia920922d8444c8216b442577c37710538ed8e919
2017-01-12 15:32:32 +00:00
Mehdi Abaakouk d38f4ccf02 replace _callables by _works
In PeriodicsTasks _callables have been renamed _works
a while ago, but one occurence was missing.

Change-Id: I16cdda70917721d4140e2217f8bfce53ee5b730d
2016-11-23 09:16:36 +01:00
Jenkins a82e6b7ef4 Merge "Add what the watcher watches to the watcher as a property" 2016-10-07 16:46:27 +00:00
Gevorg Davoian 4650a2bf3b Eliminate unneccessary patching in GreenFuture
Due to eventlet's bug eventlet.patcher.is_monkey_patched('threading')
always returns False, so it is better to replace 'threading' with
'thread', which seems to work properly.

Change-Id: I849e427c312e41b2d518a088e3f91c92d7574bd7
2016-08-09 17:15:11 +03:00
Joshua Harlow 6d4e15a676 Remove 'smart' idleness check
This needs further improvement as it does not appear
to be working as expected so for now just revert it back
to the naive (and more expected) behavior for starting up
new workers (and adding them to the worker pool).

Change-Id: Ia9cc4e44c55ea39bc6270ed42b375ca495bf9ec1
2016-07-13 13:15:46 -07:00
Joshua Harlow dc0862510f Begin adding our own thread pool executor
To be able to add extended functionality without having
to (just yet) upstream those changes into the stdlib, especially
around thread pool execution and tracking start to add our own
thread workers and the needed mechanics to clean them up
appropriately.

Change-Id: If9af0d905009435d91e7d8ac00005f4ca30bd987
2016-06-11 11:05:18 -07:00
Joshua Harlow 4d3d47d6af Add what the watcher watches to the watcher as a property
Also adds some new tests to ensure it works as expected.

Change-Id: I4cd9af710f4d528284ea7942fc70e178916bea5a
2016-03-31 23:21:37 -07:00
Jenkins 976fe90292 Merge "Use prettytable to show pretty schedule/active/planned time table" 2016-03-28 16:17:17 +00:00
Jenkins fef9387dc1 Merge "Fix time related check in rejection test" 2016-03-13 22:56:54 +00:00
Joshua Harlow 29c5d14d4a Fix 'on_failure' param not be used
Ensures it is used, and exceptions from it are
handled, and that it is type checked correctly before
being used.

Closes-Bug: #1531530

Change-Id: I0843f363ae70d3b6475cddeebb7564204a10a04d
2016-03-07 14:15:33 -08:00
Joshua Harlow e74a77671c Use prettytable to show pretty schedule/active/planned time table
Creates nice output like:

https://gist.github.com/harlowja/9e120f8c2ad7a771f8e7

Change-Id: I21627e47d093cb8f07b4fb32a71085957d1e614d
2016-03-03 09:52:28 -08:00
Joshua Harlow 9c41f41698 Fix time related check in rejection test
Instead of using time functions to delay submissions
to validate that rejection fails use an event instead
and use it to force the needed threads to block up so
that rejection will always happen (even with time
changes).

Closes-bug: #1552232

Change-Id: Ib688dccd8eda8c11b71b4aa36904ee67d2f93889
2016-03-02 17:15:12 -08:00
Joshua Harlow 74a8c7ad72 Reduce/remove duplication in run functions
Have a smaller helper class that unifies the retaining
or not retaining of failure tracebacks instead of having
two nearly equivalent functions to do the same.

Change-Id: I54bb087bbe35f5597c6658ba8aa823795f560045
2016-03-02 15:40:48 -08:00
Joshua Harlow 81991b6a09 Fix jitter strategies
Seems like the wrong function signature was being decorated
so this fixes that and adds an initial test to make sure it
continues to work as expected.

Also fixes the invalid usage of random in the jitter
decorator.

Change-Id: If30fcc1430b84078dc5fc104aa28f06471e4c358
2016-02-24 17:23:13 -08:00
Joshua Harlow bd9f7ca8e8 Single quote the callables name (when submission errors)
The rest of the code single quotes it so this code might
as well to (to follow the existing logging style).

Change-Id: Ibd2a8b49d11dcd32d9168a96950e08616ac1c54d
2016-02-22 10:08:36 +01:00
Jenkins 9cf9c74513 Merge "Reschedule failed periodic tasks after a short delay" 2016-02-22 08:06:54 +00:00
Dmitry Tantsur f9f685e51d Reschedule failed periodic tasks after a short delay
When testing Ironic with Futurist I found out that whe number of tasks
seriously exceed number of possible workers, the conductor goes into loop
retrying them and mostly stops reacting. Insert a short random delay before
retrying, so that the executor has chances of processing tasks before we
through more on it. Also recalculate the "now" value, as submitting a task
could take some time.

Change-Id: I8e45ddf7c87cf130028fc9fe937691968db17ee9
2016-02-19 10:17:33 +01:00
Dmitry Tantsur 6a2f5a0bb3 Fix wrong comparison in reject_when_reached
We say we raise when backlog goes past max size, but actually we raise when
backlogs tries to reach the max size. This patch fixes reject_when_reached to
actually allow max size to be reached, but no more.

Also adding missing unit test for reject_when_reached.

Change-Id: Ide835b6e5fd15171b9bebc7ca112b84423b39fab
2016-02-17 20:07:24 +01:00
Joshua Harlow da698e8511 Ensure all futures have completed before run returns
Now that shutdown can be skipped (which would previously
wait for all submitted futures to be finished) we need to have
a way to know that all submitted futures have been finished
before our run method returns.

A simple countdown barrier/latch does this so ensure that we use
it in the periodic run loop to be able to only return from the
run loop when no active futures are in progress.

Change-Id: Ia3dd84ebfe2416060aaf5113cc8310a23919a3f9
2016-02-09 10:03:57 -08:00
Dmitry Tantsur 7a225a3a31 Allow PeriodicWorker to skip executor shutdown in case of a preexisting executor
The current behaviour of always shutting down the executor makes it troublesome
to use a precreated executor and makes completely impossible to use the same
executor for running PeriodicWorker itself (which is what I'm doing for Ironic).

This change adds a base class ExecutorFactory with 'shutdown' attribute
defaulting to True. It also adds a new class ExistingExecutor that takes an
existing executor and sets shutdown to False.

Change-Id: I6a574ba77439aeee46b9e154aa059098ec087308
2016-02-03 11:33:10 +01:00
Joshua Harlow 588c0770e9 Expose underlying futures.CancelledError
When a future is cancelled and the result is fetched
it will raise a CancelledError which is quite useful to
be able to catch.

So to enable catching that specific exception expose it
in futurist __init__.py so that people can easily catch
it just by catching futurist.CancelledError and handling
the outcome of that.

Change-Id: I7bc1838aadcbb8cd0d635da83bb1e1b1ec761e89
2016-01-26 17:11:22 -08:00
Jenkins b11b773ddf Merge "PeriodicWorker.create to accept arguments for periodic tasks" 2016-01-16 20:23:00 +00:00
Jenkins 79eeb23680 Merge "Handle exceptions from executor.submit in PeriodicWorker" 2016-01-14 02:41:48 +00:00
Dmitry Tantsur 1f4aeed76b PeriodicWorker.create to accept arguments for periodic tasks
Sometimes it might be required to pass the same arguments to all
tasks collected by create (e.g. ironic passes context).

Change-Id: I7992c59e76b8b7b9101501d75cb7a53d982d07fd
2016-01-11 15:10:24 +01:00
Dmitry Tantsur b545f81bb9 Handle exceptions from executor.submit in PeriodicWorker
Currently PeriodicWorker fails if executor raises RejectedSubmission.
This change makes it catch RejectedSubmission and RuntimeError from
executor.submit and reschedule the task as soon as possible.

Change-Id: Ifff7850c9b44ee076f428ed3a613a13e182a549d
Closes-Bug: #1532255
2016-01-11 13:39:49 +01:00
Dmitry Tantsur 4b452427f9 Add periodics.is_periodic to check if object is a periodic task
People implementing their own periodic tasks collection (as opposed
to using PeriodicWorker.create) need to know if something is
a periodic task without inspecting its private attributes.
This function returning True means something is safe to add
to a PeriodicWorker (will not raise ValueError).

Change-Id: Iee8da601b1bc994188c58ca459574a466270bf63
2016-01-06 17:14:49 +01:00
Joshua Harlow 0a68329c50 Allow for providing different run work sync functions
It can be quite useful to replace the running of work
function in the sync executor with a function that logs
or with a function that shows whats running, or other
so enable that replacement by having a new sync executor
parameter that can be provided to give alternate functions
that run work items.

Change-Id: I8f4a806ed698370b4f5a300d74ee507cbe4eebc5
2015-10-21 16:14:51 -07:00
Joshua Harlow a72c6603a9 Have the executor stats object repr look like the watcher repr
Avoid having different repr styles and just unify the executor
stats one to look like the one used in the watcher object, which
is the style more commonly used in other libraries also.

Change-Id: Ibd98094371f57ecb8bca78b8446ca28727da84e9
2015-10-11 10:24:08 -07:00
Jenkins 24166d23da Merge "Some minor refactoring and comment/note addition" 2015-10-09 13:53:02 +00:00
Jenkins 4da07660a0 Merge "Move most of green helper classes -> '_green.py'" 2015-10-08 22:25:34 +00:00
Joshua Harlow 08f0d10a25 Add useful watcher '__repr__' function
When a workers watchers are iterated over and
say printed to stderr/stdout the default '__repr__'
doesn't provide much value; so instead of the default
provide our own that adds slightly more value.

Change-Id: I8f82a732ad3d9f0ce0ec5857d5d883b3b5f2ff1d
2015-10-07 16:52:40 -07:00
Joshua Harlow 95e9efc5f2 Some minor refactoring and comment/note addition
In the internal '_run' instead of having a loop that
has various cases with quite a bit of code per case instead
have the loop call into smaller helper functions that handle
the different types of processing the loop does.

Also adds a note to the 'reset' method that explains why
the metrics keys are overwritten vs being copied/replaced
with a new copy.

Change-Id: I5ac1794d5803850120be4e67b85efe0449aa1891
2015-10-06 22:01:37 -07:00
Joshua Harlow 711e5bb9d5 Move most of green helper classes -> '_green.py'
Instead of having the green executor(s) helper classes
be inline with the rest of the other executors move its
helper classes/objects into its own module and use them
from there instead.

This also moves the work item helper into '_utils.py' for
similar/equivalent reasons.

Change-Id: I028738bf7d64f97320450d63debbd5116b505a72
2015-09-30 16:25:12 -07:00
Dmitry Tantsur b96691eb03 Add 'enabled' parameter to @periodic decorator
This change makes @periodic more compatible with periodic tasks from
oslo.service, and also enables the following pattern found in ironic:

 @periodic(enabled=CONF.periodic_task_foo_enabled)
 def foo(...)

Change-Id: I390197158522a59145420bd7988ffa74e6e5a391
2015-09-30 11:32:17 +02:00
Joshua Harlow d537b59e45 Remove green worker reference to its executor
There really is no need to give a green worker
a reference to its creator and it hinders/makes it
harder for python to garbage collect the executor
and its workers.

It is also not used, so let's make the python
garbage collectors life easier by not even passing
it in.

Change-Id: I0f9d27663c3ca5ca908ef6f90f3d1d0f6b5f9104
2015-09-23 17:53:37 -07:00
Jenkins 7bbce9e23c Merge "Just use a deque vs reverse pushing/popping" 2015-09-16 11:57:02 +00:00
Jenkins 2ada7fba27 Merge "Ensure new entry in immediates gets processed during wait(s)" 2015-09-16 11:56:59 +00:00