The wrappertask decorator was deprecated some cycles ago[1]. This is
internal implementation so could be even removed directly.
[1] 2c58017a14
Change-Id: I0f2e25971201e2a01b3e86bd859e8a4bdb98d371
Now that we only support Python 3, use the 'yield from' keyword instead
of the @wrappertask decorator for calling co-routines from a co-routine.
This should greatly simplify backtraces when calling nested co-routines.
Change-Id: If9beaff74cf4facbc4aa4b30f31a3a087bdcad8a
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 13 of a serie of 28 patches
Change-Id: I09aa3b7ddd93087c3f92c76c893c609cb9473842
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 3 of a serie of 28 patches
Change-Id: If6e66839c128dde5fb80d90155dedb598da8d53b
The python standard library in Python 3.6.3 and earlier has a bug with
handling unhashable exceptions: https://bugs.python.org/issue28603
Although oslo_log will catch the error, make scheduler.Timeout hashable so
that all exceptions will be printable.
Prior to 640abe0c12 we just used __cmp__(),
but that isn't used in Python 3. Defining __eq__(), which is required for
the total_ordering decorator, makes the class unhashable in Python 3.
Change-Id: Idde65b2d41490ab8318b5a8b95ea74e9b96b4e5c
Related-Bug: #1724366
Related-Bug: #1721654
This changes the logic of getting cancellation grace
period of task runner before closing it: to move the
liveness check into the cancel_all() method in the
scheduler rather than ask the resource if it's IN_PROGRESS.
Change-Id: Ia2a03de227ff15cdce1b3dbb6cd6bff6c5a50a15
Partial-Bug: 1693495
Be ultra-careful to make sure that we can't end up with a local reference
to a traceback that contains the current function, thus causing a reference
loop where everything on the call stack has to be garbage collected.
Also, ignore exceptions from cancelling threads in _cancel_recursively(),
just as we do in cancel_all() since
2ffbd913a6.
Change-Id: I635c41faab4b54e95132a93f8046d0a63053485d
Related-Bug: #1626675
The commit 640abe0c12 implemented rich
comparison methods for the scheduler.Timeout class in order to be
compatible with Python3. However, it left out most of the checks to ensure
that Timeouts are compared only to other timeouts. It also left behind a
completely incorrect implmentation of __cmp__, although it didn't matter
because that is ignored for objects that implement rich comparisons.
This change deletes __cmp__, adds the type checking, and uses
functools.total_ordering to cut down on the amount of boilerplate.
Change-Id: Idf65d41705cefe9dab8ad5a518b142314077086b
Normally when exceptions are caught in DependencyTaskGroup, we cancel with
a grace period and then re-raise the first exception once all subtasks are
stopped. This means that e.g. when a resource fails, the exception the
stack gets back is from the first resource failure and not e.g. any
resulting exceptions from cancelled in-progress resources.
However, for exceptions thrown in to the task from the outside (as opposed
to raised by one of the subtasks), we generally want the *last* one to be
re-raised. Specifically, if a cancel-without-rollback exception is thrown
followed by a cancel-with-rollback, we want the with-rollback exception to
be re-raised so that the stack knows to roll back.
Change-Id: I4843f9a55a4897a7af86a9aa69df073913fd03e6
This allows the caller to specify an arbitrary function to be called after
each sleep() when running a task. Callers can use this to keep track of
progress or keep track of external state. Any exception raised by the
callback function will be thrown into the task.
Change-Id: I637166f9448169eed8696e5b0fba106b8de1547a
In the patch 5f584db8d5, we avoid calling
TaskRunner._sleep() after the first step if the wait_time is 0 or None.
This is wrong in the case of a wait_time of 0 - since this is distinct from
None, and should result in us calling eventlet.sleep(0). It's also
inconsistent in the case of wait_time=None, since for the rest of the steps
we call self._sleep(None) (and no actual sleep occurs).
This patch ensures we always call self._sleep(), passing 0 or None as
appropriate.
It also enhances the change in d66d57f187 to
ensure that we never yield from DependencyTaskGroup after all tasks are
completed, which could happen if that last batch of tasks to be started all
completed in a single step. This both saves changing the unit test to
accomodate the new _sleep(None) call and avoids an unnecessary
eventlet.sleep(0) call in real usage.
Change-Id: I301930fcba170e36d2d902c8d693ca722476cd18
This will be useful in numerous places where we use a TaskRunner to provide
a timeout when we are already inside a wrappertask. It eliminates the
need to manually advance the TaskRunner, and ensures correct exception
handling.
Change-Id: I9be0a361e1d1856adb6eee2ef0da8d5aa23795be
In some cases we may want to have different grace periods for different
tasks when we cancel a DependencyTaskGroup due to an error. This change
allows us to pass a function instead of just a value, so that we can
compute the correct grace period for each task.
Change-Id: I48842cc2a0193b8c3dc799f6e61c9243b0264b49
TaskRunner.cancel() can (and, in the case of a running Resource action with
grace_period=None, will) raise an exception. To ensure that all of the
tasks in the DependencyTaskGroup get cancelled, catch and ignore any
exceptions raised when cancelling each task.
Change-Id: I612e76d55d6f427156f6d74991e7c8f717c32200
This change allows a scheduler co-routine to yield an integer n, such
that only every nth call to TaskRunner.step() results in the co-routine
being advanced. We'll be able to use this to selectively slow down
polling of resources where and when we know it doesn't make sense to
poll every second.
Change-Id: Ifc6303524be8e477905546760279027886833617
Related-Bug: #1549219
Holding a reference to a traceback in a local variable can cause a
reference loop (since the stack frame containing the local variable is
inevitably referenced by the traceback), with the result that basically
everything may need to get garbage collected. Prevent this by explicitly
deleting the local variable holding the deletingtraceback once it is no
longer required.
Change-Id: Icde3dc0c6d5b2c799df778d3e1780514bc04bf5b
If we use unicode as resource name or parameter,
we are likely to encounter unicode problems.
This patch aims to fix unicode problems, and
there are several things to be clear:
1. For the class wrap with six.python_2_unicode_compatible, in
python2 their original __str__ function will map to __unicode__,
and new __str__ function comes from __unicode__.encode.
So we should always return unicode in __str__() if wrap
with python_2_unicode_compatible.
2. python_2_unicode_compatible will not handle __repr__,
__repr__ should return str in all versions of python.
Co-Authored-By: Ethan Lynn <xjunlin@cn.ibm.com>
Closes-Bug: #1524194
Change-Id: Ib4af43d153e0edf9d66308bf8e7b93a3be501d2e
Always catch StopIteration, rather than allowing it to bubble up a level.
This is neccessary due to PEP0479, wherein it was decided that Python will
start converting this exception to a RuntimeError when it bubbles up,
beginning with Python 3.7.
This is a follow-up to f7edd0ba2d, which
missed one case of where StopIteration could be raised.
Change-Id: I34bc32e7477bcaa0ddfe30179f1008b1be97c0a3
Some catched exception with ``except XXX as e:`` raise with exception like
``raise e``, but in this way we lose the traceback. I propose we use
``raise`` to keep traceback.
Change-Id: I89fe43ef1af5374e4b2c4151ddc4b2d4f7729601
For an update that involves e.g. stepping through the whole graph and
verifying that nothing needs to be updated, we spend a lot of time sleeping
unnecessarilty. Every task will exit without yielding (i.e.
it will be complete after calling TaskRunner.start()), yet the
DependencyTaskGroup yields after each set of tasks so the minimum sleep
time in seconds is the maximum path length in the graph minus one.
This change fixes that by removing nodes from the graph immediately if they
are done immediately after having been started. Since the _ready() call
returns an iterator, this allows any later tasks that were blocking only on
this one to start immediately. To ensure that any tasks that are only
blocking on this one _do_ appear later, iterate over the graph in
topologically sorted order.
The potential downside to this would be any time that actions complete
quickly (i.e. without yielding), but we still need to throttle them. An
obvious case might be a resource type with no check_create_complete()
function - creating a lot of these in a row could result in quota failures
on the target API. However, the Resource.action_handler_task() task always
yields at least once even if there is no check, so this patch should not
change its behaviour.
Change-Id: I734561814d2784e710d0b9ec3ef7834f44f579b2
Closes-Bug: #1523303
Ensure that an unbound method is never logged as "<method name> from None".
Also report the actual action name when doing stack_action().
Change-Id: Iff37860525d182e6eee01a7cc52c6599f05bff8c
Related-Bug: #1339759
The process of bringing us into compliance with the H405 pep8 rule has left
us with a lot of docstrings that either don't follow the recommendations of
PEP0257, are misleading, wrong, or nonsensical. This patch attempts to fix
them up.
Change-Id: Icfe90b5ba3102a7e13ab659038a8b2af9019e9e6
Always catch StopIteration, rather than allowing it to bubble up a level.
This is neccessary due to PEP0479, wherein it was decided that Python will
start converting this exception to a RuntimeError when it bubbles up,
beginning with Python 3.7.
Change-Id: I4e93b0e85500455de415188cc679961035958bd1
Always use a def statement instead of an assignment statement that binds
a lambda expression directly to an identifier.
Yes:
def f(x): return 2*x
No:
f = lambda x: 2*x
https://www.python.org/dev/peps/pep-0008/
Change-Id: I776b80bb4bc2184125f461c801897f84d3f29254
We're close to the end of porting all unit tests
to be compatible with python34. However, ~8 tests are skipped
specifically in python34 because of bugs mostly in mox3.
Those tests will likely have to be converted to mock in
the future. Note that from now on, every test added will have
to be python34 compatible.
Also, fix __str__ function of ExceptionGroup in test_scheduler.
It intentionally uses str so as to not flatten the unicode
literal in python2.7 and also be compatible with python3.4
blueprint heat-python34-support
Change-Id: Id9b8aa11b17c96895092dd701b761a479df09282
Use volume progress objects instead (as in volume attachment resources).
Module volume_tasks and class scheduler.PollingTaskGroup have been removed,
as they are no longer used anywhere.
Change-Id: Iff3b92f28d89ad1fdfd22511bb531be463855ccd
Partial-Bug: #1393268
There are three tests skipped which needs further
addressing later on. However, fixing test_stack should prevent
future regressions for when a stack object is created.
partial blueprint heat-python34-support
Change-Id: I15bfd0990ebe248436c1a0c5e3f4f9a8332c2a57
On Python 3, the decorator does nothing. On Python 2, it aliases the
__str__ method to __unicode__ and creates a new __str__ method that
returns the result of __unicode__() encoded with UTF-8.
All instances of __unicode__ have been removed with the
aforementioned decorator in place instead.
Note: There are some instances of __str__ which aren't handled but
will be once we have actual python34 tests running in the gate.
partial blueprint heat-python34-support
Change-Id: I7271a2581e1c2bbc282933c7da73db810c7e09db
__nonzero__ has been renamed to __bool__ in python3.4. So add the
__bool__ magic method that will in turn call the existing __nonzero__
method. This avoids the scheduler going into an infinte loop when
it checks for the no. of tasks remaining.
partial blueprint heat-python34-support
Change-Id: I5de554805a2af9abc15808427ce489720287f6b3
* Use six.moves.reduce
* Update numliteral from 0L to 0
* Use open() instead of file()
* Use rich comparison methods instead of cmp()
partial blueprint heat-python34-support
Change-Id: I9ffd85645563192b5d6124f8dbb71c24245eefc4
This results in a functional test run which is up to 20% faster than
the current sleep(1)
Closes-Bug: #1439216
Change-Id: I1f594963dd164e44b8d9c601610a4036a0253ece
Simply adding an extra colon in the from_task_with_args docstring to
make the examples show up as code in the rendered documentation.
Co-Author: Qiming Teng
Change-Id: Ia1704a2f2517e2de2fb4f300979f63fbec0e9f7a