Commit Graph

91 Commits

Author SHA1 Message Date
Takashi Kajinami f5e5aaed8e Remove deprecated wrappertask decorator
The wrappertask decorator was deprecated some cycles ago[1]. This is
internal implementation so could be even removed directly.

[1] 2c58017a14

Change-Id: I0f2e25971201e2a01b3e86bd859e8a4bdb98d371
2023-12-27 09:06:33 +09:00
Zane Bitter 2c58017a14 Deprecate wrappertask decorator
Now that we only support Python 3, use the 'yield from' keyword instead
of the @wrappertask decorator for calling co-routines from a co-routine.
This should greatly simplify backtraces when calling nested co-routines.

Change-Id: If9beaff74cf4facbc4aa4b30f31a3a087bdcad8a
2020-04-30 09:56:20 -04:00
Hervé Beraud 5fa48d67a2 Remove six and python 2.7 full support
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.

Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.

To review this patch you can use the six documentation [1] to obtain help and
understand choices.

Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.

Latin-1 is equal to iso-8859-1 [3].

This encoding is the default encoding [4] of certain descriptive HTTP
headers.

I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).

HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].

Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'

[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp

Patch 13 of a serie of 28 patches

Change-Id: I09aa3b7ddd93087c3f92c76c893c609cb9473842
2020-04-23 14:49:12 +02:00
Hervé Beraud fccd312871 Remove six and python 2.7 full support
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.

Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.

To review this patch you can use the six documentation [1] to obtain help and
understand choices.

Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.

Latin-1 is equal to iso-8859-1 [3].

This encoding is the default encoding [4] of certain descriptive HTTP
headers.

I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).

HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].

Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'

[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp

Patch 3 of a serie of 28 patches

Change-Id: If6e66839c128dde5fb80d90155dedb598da8d53b
2020-04-22 12:23:44 +02:00
Andreas Jaeger 57eb87149f Fix hacking warnings
Fix some warnings found by hacking and enable them again.

Change-Id: Ia09de4d0fda752a009b9246b4e6d485601cd9562
2020-04-16 06:43:27 +00:00
Zane Bitter 6a9672a264 Make scheduler.Timeout exception hashable
The python standard library in Python 3.6.3 and earlier has a bug with
handling unhashable exceptions: https://bugs.python.org/issue28603

Although oslo_log will catch the error, make scheduler.Timeout hashable so
that all exceptions will be printable.

Prior to 640abe0c12 we just used __cmp__(),
but that isn't used in Python 3. Defining __eq__(), which is required for
the total_ordering decorator, makes the class unhashable in Python 3.

Change-Id: Idde65b2d41490ab8318b5a8b95ea74e9b96b4e5c
Related-Bug: #1724366
Related-Bug: #1721654
2017-10-18 16:46:39 -04:00
huangtianhua 8c62e96947 Get cancellation grace period correctly
This changes the logic of getting cancellation grace
period of task runner before closing it: to move the
liveness check into the cancel_all() method in the
scheduler rather than ask the resource if it's IN_PROGRESS.

Change-Id: Ia2a03de227ff15cdce1b3dbb6cd6bff6c5a50a15
Partial-Bug: 1693495
2017-06-29 01:47:08 +00:00
liyi 8f10215ffd Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.

See:
http://lists.openstack.org/pipermail/openstack-i18n/2016-November/002574.html
http://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html

Change-Id: Ieec8028305099422e1b0f8fc84bc90c9ca6c694f
2017-03-25 17:11:50 +08:00
Luong Anh Tuan a6f34dd1af Avoid Forcing the Translation of Translatable Variables
Whenever possible translation should not be forced by use of six.text_type()
on a message being used with a format string.

http://docs.openstack.org/developer/oslo.i18n/guidelines.html#avoid-forcing-the-translation-of-translatable-variables

Change-Id: Iabde2f7d6a98115c0bf7d3ef097bafb0a13d6cbe
2016-12-20 04:07:12 +00:00
Zane Bitter 38483c56fc Avoid circular refs more aggressively in DependencyTaskGroup
Be ultra-careful to make sure that we can't end up with a local reference
to a traceback that contains the current function, thus causing a reference
loop where everything on the call stack has to be garbage collected.

Also, ignore exceptions from cancelling threads in _cancel_recursively(),
just as we do in cancel_all() since
2ffbd913a6.

Change-Id: I635c41faab4b54e95132a93f8046d0a63053485d
Related-Bug: #1626675
2016-09-22 18:24:05 -04:00
Jenkins ddf61d8bda Merge "Raise the last exception thrown into DependencyTaskGroup" 2016-07-20 14:17:25 +00:00
Jenkins 94a8d440c6 Merge "Scheduler: Add a progress callback to TaskRunner" 2016-07-20 14:17:09 +00:00
Jenkins c9c29d5f23 Merge "Always call TaskRunner._sleep()" 2016-07-20 14:17:00 +00:00
Jenkins 04ebd7e66a Merge "Scheduler: Allow TaskRunner to be used as a task" 2016-07-20 13:53:37 +00:00
Zane Bitter a23928ce80 Implement the correct total ordering for scheduler.Timeout
The commit 640abe0c12 implemented rich
comparison methods for the scheduler.Timeout class in order to be
compatible with Python3. However, it left out most of the checks to ensure
that Timeouts are compared only to other timeouts. It also left behind a
completely incorrect implmentation of __cmp__, although it didn't matter
because that is ignored for objects that implement rich comparisons.

This change deletes __cmp__, adds the type checking, and uses
functools.total_ordering to cut down on the amount of boilerplate.

Change-Id: Idf65d41705cefe9dab8ad5a518b142314077086b
2016-07-18 20:37:44 +00:00
Zane Bitter 796cea6cf4 Raise the last exception thrown into DependencyTaskGroup
Normally when exceptions are caught in DependencyTaskGroup, we cancel with
a grace period and then re-raise the first exception once all subtasks are
stopped. This means that e.g. when a resource fails, the exception the
stack gets back is from the first resource failure and not e.g. any
resulting exceptions from cancelled in-progress resources.

However, for exceptions thrown in to the task from the outside (as opposed
to raised by one of the subtasks), we generally want the *last* one to be
re-raised. Specifically, if a cancel-without-rollback exception is thrown
followed by a cancel-with-rollback, we want the with-rollback exception to
be re-raised so that the stack knows to roll back.

Change-Id: I4843f9a55a4897a7af86a9aa69df073913fd03e6
2016-07-18 16:26:17 -04:00
Zane Bitter 991d41f255 Scheduler: Add a progress callback to TaskRunner
This allows the caller to specify an arbitrary function to be called after
each sleep() when running a task. Callers can use this to keep track of
progress or keep track of external state. Any exception raised by the
callback function will be thrown into the task.

Change-Id: I637166f9448169eed8696e5b0fba106b8de1547a
2016-07-18 16:26:17 -04:00
Zane Bitter 1de87a9b36 Always call TaskRunner._sleep()
In the patch 5f584db8d5, we avoid calling
TaskRunner._sleep() after the first step if the wait_time is 0 or None.
This is wrong in the case of a wait_time of 0 - since this is distinct from
None, and should result in us calling eventlet.sleep(0). It's also
inconsistent in the case of wait_time=None, since for the rest of the steps
we call self._sleep(None) (and no actual sleep occurs).

This patch ensures we always call self._sleep(), passing 0 or None as
appropriate.

It also enhances the change in d66d57f187 to
ensure that we never yield from DependencyTaskGroup after all tasks are
completed, which could happen if that last batch of tasks to be started all
completed in a single step. This both saves changing the unit test to
accomodate the new _sleep(None) call and avoids an unnecessary
eventlet.sleep(0) call in real usage.

Change-Id: I301930fcba170e36d2d902c8d693ca722476cd18
2016-07-18 16:26:17 -04:00
Zane Bitter ecdaa8e183 Scheduler: Allow TaskRunner to be used as a task
This will be useful in numerous places where we use a TaskRunner to provide
a timeout when we are already inside a wrappertask. It eliminates the
need to manually advance the TaskRunner, and ensures correct exception
handling.

Change-Id: I9be0a361e1d1856adb6eee2ef0da8d5aa23795be
2016-07-18 16:26:16 -04:00
Zane Bitter 1163e4695e Scheduler: Allow a per-task customisable grace period
In some cases we may want to have different grace periods for different
tasks when we cancel a DependencyTaskGroup due to an error. This change
allows us to pass a function instead of just a value, so that we can
compute the correct grace period for each task.

Change-Id: I48842cc2a0193b8c3dc799f6e61c9243b0264b49
2016-07-13 17:09:42 -04:00
Zane Bitter 2ffbd913a6 Catch exceptions when cancelling tasks in DependencyTaskGroup
TaskRunner.cancel() can (and, in the case of a running Resource action with
grace_period=None, will) raise an exception. To ensure that all of the
tasks in the DependencyTaskGroup get cancelled, catch and ignore any
exceptions raised when cancelling each task.

Change-Id: I612e76d55d6f427156f6d74991e7c8f717c32200
2016-07-13 17:09:42 -04:00
Zane Bitter 8f8330573b Allow scheduler tasks to request longer wait periods
This change allows a scheduler co-routine to yield an integer n, such
that only every nth call to TaskRunner.step() results in the co-routine
being advanced. We'll be able to use this to selectively slow down
polling of resources where and when we know it doesn't make sense to
poll every second.

Change-Id: Ifc6303524be8e477905546760279027886833617
Related-Bug: #1549219
2016-05-17 12:29:12 -04:00
Zane Bitter e79f28d28a Fix unfortunate spelling error
Change-Id: I2ceab6579112d55f496c0bbe37c42263b54bb0c9
2016-04-04 12:43:16 -04:00
Zane Bitter 2897634814 Avoid reference loops in DependencyTaskGroup
Holding a reference to a traceback in a local variable can cause a
reference loop (since the stack frame containing the local variable is
inevitably referenced by the traceback), with the result that basically
everything may need to get garbage collected. Prevent this by explicitly
deleting the local variable holding the deletingtraceback once it is no
longer required.

Change-Id: Icde3dc0c6d5b2c799df778d3e1780514bc04bf5b
2016-03-30 14:29:47 -04:00
Ethan Lynn 6cf8dc6e56 Fix problems in unicode template support
If we use unicode as resource name or parameter,
we are likely to encounter unicode problems.
This patch aims to fix unicode problems, and
there are several things to be clear:

1. For the class wrap with six.python_2_unicode_compatible, in
python2 their original __str__ function will map to __unicode__,
and new __str__ function comes from __unicode__.encode.
So we should always return unicode in __str__() if wrap
with python_2_unicode_compatible.

2. python_2_unicode_compatible will not handle __repr__,
__repr__ should return str in all versions of python.

Co-Authored-By: Ethan Lynn <xjunlin@cn.ibm.com>
Closes-Bug: #1524194
Change-Id: Ib4af43d153e0edf9d66308bf8e7b93a3be501d2e
2016-03-04 01:59:40 +00:00
Zane Bitter 3c96f5a084 Fix scheduler compatibility with Python 3.7 harder
Always catch StopIteration, rather than allowing it to bubble up a level.
This is neccessary due to PEP0479, wherein it was decided that Python will
start converting this exception to a RuntimeError when it bubbles up,
beginning with Python 3.7.

This is a follow-up to f7edd0ba2d, which
missed one case of where StopIteration could be raised.

Change-Id: I34bc32e7477bcaa0ddfe30179f1008b1be97c0a3
2016-02-10 22:52:48 -05:00
ricolin 89abb7017d Raise with traceback
Some catched exception with ``except XXX as e:`` raise with exception like
``raise e``, but in this way we lose the traceback. I propose we use
``raise`` to keep traceback.

Change-Id: I89fe43ef1af5374e4b2c4151ddc4b2d4f7729601
2016-01-14 01:12:32 +00:00
Jenkins 19d20e131b Merge "Improve log messages from scheduler" 2015-12-08 22:32:19 +00:00
Zane Bitter d66d57f187 Eliminate unnecessary sleeps during no-op update
For an update that involves e.g. stepping through the whole graph and
verifying that nothing needs to be updated, we spend a lot of time sleeping
unnecessarilty. Every task will exit without yielding (i.e.
it will be complete after calling TaskRunner.start()), yet the
DependencyTaskGroup yields after each set of tasks so the minimum sleep
time in seconds is the maximum path length in the graph minus one.

This change fixes that by removing nodes from the graph immediately if they
are done immediately after having been started. Since the _ready() call
returns an iterator, this allows any later tasks that were blocking only on
this one to start immediately. To ensure that any tasks that are only
blocking on this one _do_ appear later, iterate over the graph in
topologically sorted order.

The potential downside to this would be any time that actions complete
quickly (i.e. without yielding), but we still need to throttle them. An
obvious case might be a resource type with no check_create_complete()
function - creating a lot of these in a row could result in quota failures
on the target API. However, the Resource.action_handler_task() task always
yields at least once even if there is no check, so this patch should not
change its behaviour.

Change-Id: I734561814d2784e710d0b9ec3ef7834f44f579b2
Closes-Bug: #1523303
2015-12-07 18:03:39 -05:00
Zane Bitter 242b06664f Improve log messages from scheduler
Ensure that an unbound method is never logged as "<method name> from None".

Also report the actual action name when doing stack_action().

Change-Id: Iff37860525d182e6eee01a7cc52c6599f05bff8c
Related-Bug: #1339759
2015-12-04 20:58:18 -05:00
Zane Bitter 388ee0257f Fix garbled docstrings
The process of bringing us into compliance with the H405 pep8 rule has left
us with a lot of docstrings that either don't follow the recommendations of
PEP0257, are misleading, wrong, or nonsensical. This patch attempts to fix
them up.

Change-Id: Icfe90b5ba3102a7e13ab659038a8b2af9019e9e6
2015-11-10 11:12:36 -05:00
Zane Bitter f7edd0ba2d Fix scheduler compatibility with Python 3.7
Always catch StopIteration, rather than allowing it to bubble up a level.
This is neccessary due to PEP0479, wherein it was decided that Python will
start converting this exception to a RuntimeError when it bubbles up,
beginning with Python 3.7.

Change-Id: I4e93b0e85500455de415188cc679961035958bd1
2015-11-10 08:41:52 -05:00
Jenkins 2df093b75e Merge "py34: Get rid of py3-testlist" 2015-09-25 20:45:16 +00:00
Dmitry Tyurnikov 1bb37b6857 Some lambda were replaced on def
Always use a def statement instead of an assignment statement that binds
a lambda expression directly to an identifier.

Yes:
def f(x): return 2*x

No:
f = lambda x: 2*x

https://www.python.org/dev/peps/pep-0008/

Change-Id: I776b80bb4bc2184125f461c801897f84d3f29254
2015-09-23 11:35:29 +01:00
Sirushti Murugesan 3f635a26a4 py34: Get rid of py3-testlist
We're close to the end of porting all unit tests
to be compatible with python34. However, ~8 tests are skipped
specifically in python34 because of bugs mostly in mox3.
Those tests will likely have to be converted to mock in
the future. Note that from now on, every test added will have
to be python34 compatible.

Also, fix __str__ function of ExceptionGroup in test_scheduler.
It intentionally uses str so as to not flatten the unicode
literal in python2.7 and also be compatible with python3.4

blueprint heat-python34-support

Change-Id: Id9b8aa11b17c96895092dd701b761a479df09282
2015-09-22 18:13:29 +00:00
Peter Razumovsky 2da170c435 Fix [H405] pep rule in heat/engine
Fix [H405] rule in heat/engine python
files.

Implements bp docstring-improvements

Change-Id: Iaa1541eb03c4db837ef3a0e4eb22393ba32e270f
2015-09-21 14:51:46 +03:00
Pavlo Shchelokovskyy 4f9371cfc0 Do not use VolumeTasks in AWS Instance create
Use volume progress objects instead (as in volume attachment resources).

Module volume_tasks and class scheduler.PollingTaskGroup have been removed,
as they are no longer used anywhere.

Change-Id: Iff3b92f28d89ad1fdfd22511bb531be463855ccd
Partial-Bug: #1393268
2015-07-28 21:59:06 +00:00
Jenkins 4818778a53 Merge "Get heat.tests.test_stack* working in py34 env" 2015-07-22 04:50:55 +00:00
Rabi Mishra 8b6e1eac1d Refactor timeout calculation to utility
Change-Id: I8d846a2c99a9bd88bbff8115646f238a6244de8a
2015-07-18 08:51:59 +05:30
Sirushti Murugesan 14014a6989 Get heat.tests.test_stack* working in py34 env
There are three tests skipped which needs further
addressing later on. However, fixing test_stack should prevent
future regressions for when a stack object is created.

partial blueprint heat-python34-support

Change-Id: I15bfd0990ebe248436c1a0c5e3f4f9a8332c2a57
2015-07-14 14:10:51 +05:30
Sirushti Murugesan 2c99f0090c Adopt @six.python_2_unicode_compatible decorator
On Python 3, the decorator does nothing. On Python 2, it aliases the
__str__ method to __unicode__ and creates a new __str__ method that
returns the result of __unicode__() encoded with UTF-8.

All instances of __unicode__ have been removed with the
aforementioned decorator in place instead.

Note: There are some instances of __str__ which aren't handled but
will be once we have actual python34 tests running in the gate.

partial blueprint heat-python34-support

Change-Id: I7271a2581e1c2bbc282933c7da73db810c7e09db
2015-07-09 12:44:03 +05:30
Sirushti Murugesan f1f4d60e0a Add __bool__ for classes that implement __nonzero__
__nonzero__ has been renamed to __bool__ in python3.4. So add the
__bool__ magic method that will in turn call the existing __nonzero__
method. This avoids the scheduler going into an infinte loop when
it checks for the no. of tasks remaining.

partial blueprint heat-python34-support

Change-Id: I5de554805a2af9abc15808427ce489720287f6b3
2015-07-02 07:56:31 +00:00
Sirushti Murugesan 640abe0c12 Update the rest of the code to satisfy flake8 in a py34 env
* Use six.moves.reduce
* Update numliteral from 0L to 0
* Use open() instead of file()
* Use rich comparison methods instead of cmp()

partial blueprint heat-python34-support

Change-Id: I9ffd85645563192b5d6124f8dbb71c24245eefc4
2015-04-24 11:53:43 +05:30
Sirushti Murugesan 796680c9c7 Use six.text_type and six.reraise
partial blueprint heat-python34-support

Change-Id: Idce66792a43368c78002683cf8cf1d8759729658
2015-04-24 11:53:43 +05:30
Sirushti Murugesan 8257306624 Use six.moves.map/filter/zip
partial blueprint heat-python34-support

Change-Id: Iec4d228933d5cdc2474a8090cc524673d0ecdb2c
2015-04-24 11:52:39 +05:30
Sirushti Murugesan 63ef40f1be Update xrange and lambda for python3
partial blueprint heat-python34-support

Change-Id: I0b4ebf640083e63d6d1d461bf0c97778d19d770e
2015-04-23 19:45:36 +05:30
Tetiana Lashchova 4c6810ce1a Use six.itervalues(x) instead of x.values()
Change-Id: Ie71072940b7b5e599d9012be0bc1874b2a623003
Partial-Bug: #1446254
2015-04-21 09:05:49 +03:00
Steve Baker 5f584db8d5 Do sleep(0) for first task sleep
This results in a functional test run which is up to 20% faster than
the current sleep(1)

Closes-Bug: #1439216
Change-Id: I1f594963dd164e44b8d9c601610a4036a0253ece
2015-04-02 11:34:29 +13:00
Jenkins 7feaa881eb Merge "Add code tag for from_task_with_args() docstring" 2015-03-02 01:04:43 +00:00
matts2006 731c290df7 Add code tag for from_task_with_args() docstring
Simply adding an extra colon in the from_task_with_args docstring to
make the examples show up as code in the rendered documentation.

Co-Author: Qiming Teng
Change-Id: Ia1704a2f2517e2de2fb4f300979f63fbec0e9f7a
2015-02-28 12:20:27 +08:00