Now that we only support Python 3, use the 'yield from' keyword instead
of the @wrappertask decorator for calling co-routines from a co-routine.
This should greatly simplify backtraces when calling nested co-routines.
Change-Id: If9beaff74cf4facbc4aa4b30f31a3a087bdcad8a
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 14 of a serie of 28 patches
Change-Id: Iccb743056c631f2e0728013942d764cd8b1ecfe0
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 3 of a serie of 28 patches
Change-Id: If6e66839c128dde5fb80d90155dedb598da8d53b
At the start of a legacy stack update, we calculate the frozen definitions
for all of the existing resources. This is so that we can later compare
them against the new resources to see if they need to be updated.
Since resources that are being deleted (i.e. don't appear in the new
template) won't ever need to be compared to the new definitions, there's no
need for us to calculate this data.
Apparently there are (extremely rare) situations where getting the frozen
definitions will fail, so this at least gives users an opportunity to
recover by deleting only the resources causing trouble rather than the
whole stack.
Change-Id: I04bed83cc2181e1e61bf90c2f259cfd123362b06
Story: 2002966
Task: 22972
Since we remove a resource's definition after it is deleted, we must also
update a resource's definition after it is updated. Otherwise, a resource
that depended on the now-deleted resource prior to its last in-place update
could remain in the template, leaving the stack inoperable since it is
unable to calculate its own dependencies.
We already update the template in the backup stack when updating resources
in-place, and in the main stack when creating replacement resources. Also
update the main template when resources are updated in-place, not just when
they are replaced.
Change-Id: I97ddc6a970ffd27bfd40bb8f6bec8d1cbf28a815
Story: #2003612
Task: 24946
Add converge parameter for stack update API and RPC call,
that allow triggering observe on reality. This will be
triggered by API call with converge argument (with True
or False value) within. This flag also works for resources
within nested stack.
Implements bp get-reality-for-resources
Change-Id: I151b575b714dcc9a5971a1573c126152ecd7ea93
This unifies the 'lightweight stack' used in convergence with the how
things work the rest of the time: we now always obtain resource data from
the StackDefinition's ResourceProxy objects. This means that when we are
checking an individual resource, we will never create all of the other
Resource objects for the stack in memory (although we already avoided
loading data for them from the database) - now we will only create
ResourceProxy objects as needed.
Change-Id: Id7472557e26d172df88841ff7f20afdd7f5bfada
Implements: blueprint stack-definition
Since function.dep_attrs() returns logical resource names (rather
than actual objects), we can just as easily use the StackDefinition to
calculate it instead of the Stack and Resource objects.
In the legacy path, we must ensure we use the StackDefinition from the
*new* stack to determine which attributes to include in the NodeData, since
that's what we're going to be using it for. In the convergence path the
current stack definition already contains the new template.
Also, update the *new* stack's definition with the NodeData obtained from
completed resources (in addition to the existing stack's), so that that
data may be used in calculating the dep_attrs for future resources. This is
required when get_attr functions are nested in the template.
Change-Id: I23efcc091eae53470f7f9cb3ca21e09f00f43808
Partially-Implements: blueprint stack-definition
In convergence stacks, we obtain the NodeData for the resources the current
resource depends on from the RPC message triggering the resource check. For
the legacy path, this change updates the NodeData in the StackDefinition
after each resource is created or updated.
This means that the legacy and convergence paths will be able to access
data about other resources in essentially the same way.
Change-Id: I61c61745d864de99b0127f4103a6c3a83379c516
Partially-Implements: blueprint stack-definition
We were using the definition of the existing resource, immediately after
updating it in place, to copy to the backup stack. This turns out to be
incorrect, because the definition of the new resource is only copied to the
existing resource if it was actually updated. It doesn't follow that the
existing resource's definition must be the same, because it's actually the
frozen definition (using the stored property values from the DB) that's
compared with the new definition when deciding to update. The existing
definition might well be rubbish if a previous update is failed, because it
does not get updated in the existing template after the resource is updated
in place. This caused bug 1622795 when combined with the new conditionals
feature.
We were also using the wrong template to do the parsing in both cases: when
updating in-place we are passing the new definition, but we were reparsing
using the existing template. And we were effectively doing the same when
creating a new resource, because we accessed the template through the new
resource only *after* it had been moved to the existing stack. In both
cases the correct code would have been:
definition = new_res.t.reparse(self.previous_stack,
new_res.stack.t)
However, there's no actual need to reparse the resource definitions before
passing them to Template.add_resource(), because that method immediately
calls render_hot() to get back to the unparsed definition anyway. So just
don't bother. These issues have been requiring us up until now to cache the
conditionals from the new_stack in the Template, to prevent the reparsing
from looking at the wrong set of conditionals. This change relieves us of
that restriction.
Finally, change the debug error messages to distinguish this case (copying
a resource definition into the backup template) from "backing up" a
resource (moving an actual physical resource into the backup stack).
Change-Id: I7be92f2e1b812c23fa52d87c18c7f22f1be94446
Closes-Bug: #1622795
If you attempt to update a stack containing
OS::Heat::SoftwareDeployments resources, so it uses the new
non-deprecated OS::Heat::SoftwareDeploymentGroup type instead, it
deletes the group, and all of the deployments.
This means that any deployment "actions" property will be
misinterpreted, e.g if you have actions: CREATE, all the deployments
will re-run on the update, even though it's an update, not a create.
This issue exists on all deprecated resoruces, when we trying to upgrade
to new version of it by update.
This patch fix above update issue by check if resoruce was deprecated
and been update by replacing resource (which is the parent class of
existing resource).
Change-Id: Ib7880120a90c4497a7ceea53eee55c220a28d14e
Closes-Bug: #1528958
A deeply misguided effort to move all exceptions out of the
heat.engine.resource module, where they belong, and into the
heat.common.exception module, where they largely do not, broke the API for
third-party resource plugins. Unfortunately this happened a couple of
releases back already, so we can't simply put UpdateReplace back where it
belongs as that would also break the de-facto third-party API.
This change adds an alias in the correct location and a comment indicating
that it will move back at some time in the future. It also switches all of
the in-tree uses back to heat.engine.resource.UpdateReplace, in the hope
that third-party developers will be more inclined to copy from that.
This reverts commit 4e2cfb991a.
Change-Id: Iedd5d07d6c0c07e39e51a0fb810665b3e9c61f87
Closes-Bug: #1611104
Use a progress callback for cancelling updates so that we will apply the
usual grace period to allow resources to complete when cancelling a stack
update, instead of cancelling immediately. Previously, resources could
unneccessarily be marked as FAILED, resulting in them unneccessarily being
replaced on a subsequent update.
Change-Id: I0a5bf998be764e4d6d4c6ea86aaca7f715b93d06
Related-Bug: #1591337
Instead of having a single (configurable) error_wait_time to wait after
errors before cancelling other resources in the same stack, allow each
resource to specify the grace period.
By default, IN_PROGRESS resources return error_wait_time as before, unless
they have implemented a handle_<action>_cancel() for the current action. In
this case there is no grace period; the task will be cancelled and
handle_<action>_cancel() called immediately.
Change-Id: I94255fdeadfaa398cdd34a6f83ce5ed1a7890540
If we use unicode as resource name or parameter,
we are likely to encounter unicode problems.
This patch aims to fix unicode problems, and
there are several things to be clear:
1. For the class wrap with six.python_2_unicode_compatible, in
python2 their original __str__ function will map to __unicode__,
and new __str__ function comes from __unicode__.encode.
So we should always return unicode in __str__() if wrap
with python_2_unicode_compatible.
2. python_2_unicode_compatible will not handle __repr__,
__repr__ should return str in all versions of python.
Co-Authored-By: Ethan Lynn <xjunlin@cn.ibm.com>
Closes-Bug: #1524194
Change-Id: Ib4af43d153e0edf9d66308bf8e7b93a3be501d2e
Restricts update/replace of resource based on
environment 'restricted_actions' entry for a resource.
This also adds a 'preview_update' method to Resource
class which is now leveraged by stack-preview.
Change-Id: Iea80a6572aa696ee607682e5113204d1c82389fa
Blueprint: stack-update-restrict
Currently the update preview code has no support for previewing
the effect of an update on nested stacks, which I assume was an
oversight in the original implementation. So this adds a show_nested
flag to the API which allows enabling recursive preview of the whole
update including nested stacks.
Closes-Bug: #1521971
Depends-On: I06f3b52d5d48dd5e6e266321e58ca8e6116d6017
Change-Id: I96af4d2f07056846aac7ae9ad9b6eb160e8bd51a
Currently the update preview code only considers _needs_update, but
the actual update code does a type comparison which is used to decide
whether to attempt an in-place update vs a replacement.
Thus, we currently erroneously say a resource is either unchanged or
updated during update --dry-run when the type changes, but during
the actual update we replace the resource. So instead use the same
type comparison used during update to provide a more accurate preview.
Change-Id: Idb2d9a55d47748007a438f84bb1f2259e553c067
Partial-Bug: #1521971
Now that we have get_class_to_instantiate() and it always returns
TemplateResource (instead of a custom subclass) for template resources, we
can just compare the types of resources directly.
Change-Id: I96275c3ead3ea3565e55b99a8c5a9948eaa15b78
Related-Bug: #1508115
Related-Bug: #1447194
Related-Bug: #1518458
We previously only allowed a resource to update in-place when the 'type'
field in the old and new templates matched. In an age of environment type
mappings, this is incorrect. Instead, allow a resource to be updated
in-place when the old and new resource plugins are of the same class, since
this is actually what determines whether handle_update is capable of
correctly updating the resource in-place.
There is a special-case for TemplateResource as we currently subclass it
for each template type, but all TemplateResource subclasses are in fact
capable of converting between each other via a stack update.
Change-Id: Iba4abf5efd9737ca6a17d8c91d5c54ab6d3f12e0
Co-Authored-By: Zane Bitter <zbitter@redhat.com>
Closes-Bug: #1508115
The process of bringing us into compliance with the H405 pep8 rule has left
us with a lot of docstrings that either don't follow the recommendations of
PEP0257, are misleading, wrong, or nonsensical. This patch attempts to fix
them up.
Change-Id: Icfe90b5ba3102a7e13ab659038a8b2af9019e9e6
It is convenient to have all exceptions in exception module.
Also it is reduces namespace cluttering of resource module and decreases
the number of dependencies in other modules (we do not need to import resource
in some cases for now).
UpdateReplace exception is moved in this patch.
Change-Id: Ief441ca2022a0d50e88d709d1a062631479715b7
Allow users to see what resources will be changed during a stack-update.
Docs change here https://review.openstack.org/132870/
Client change here https://review.openstack.org/#/c/126957/
BP: update-dry-run
Co-Authored-By: Jason Dunsmore <jasondunsmore@gmail.com>
Change-Id: If58bdcccfef6f5d36c0367c5267f95014232015e
The root cause came discussion in
https://review.openstack.org/#/c/175868/.
It is better to copy a new resources that appears during
stack-update to backup stack as soon as possible because
it reduces time when backup stack is not synchronized with
the existing stack.
Change-Id: If104037225905dc9c504972864270d6e68e08d73
The patch changes the approach of resource backup during stack
update. It stores the definition of resource that needs to be
updated in-place to backup stack(if it was not created before).
Without this functionality, InvalidTemplateReference will be thrown
every time when update-replaced resource has a reference to updated
in-place resource and stack update was failed. In this case, backup
stack has a resource that refers to not-presented another resource
and backup stack deletion generates an exception.
Change-Id: I436c44a579bb4df3031d1a17b9ca5b62da37afeb
Closes-bug: #1446575
Different type of resource with same name should consider to be a
new resource when stack-update.
Change-Id: I102ffbd1a1562c164f43969c87570096c280738a
Closes-Bug: #1404201
The oslo-incubator log modlule has been removed, so port to the oslo_log
library. Note this uses the new (non namespaced, e.g oslo.log) import
convention, we'll need to align other imports in a future commit.
Some import reordering was required due to pedantic H30[57] checks, and
the services have all been converted to initialize the oslo_log library
as this is done differently to the log.py in incubator.
Change-Id: Ib5a97123fe1b287bc531e42d7887c13ba6205628
When an error occurs during a stack update operation,
don't immediately stop the operation. Instead,
wait up to 4 minutes for resources already in progress
to complete before forcibly cancelling them.
Change-Id: Ib089dbc675c519668a44d1b0e3214feca5079978
Closes-bug: #1381984
Currently, Log translation is motivated by oslo's move to prioritized
translation of strings, as documented at
https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation
- add log translation hints for warning, error and info levels
- move from LOG.warning to LOG.warn
- remove use of % as a string formatter, use the log functionality
instead
Partial implements blueprint log-translation-hints
Change-Id: Ib91983e6e61b4d6ddbbed0f95dc98cbde1d4f673
Implements cancelling of stack update in progress, for any stack
regardless of disable_rollback status.
As a side-effect, a framework for sending arbitrary one-time signals from
outside into a task running as greenthread is added. ThreadgroupManager
holds references of greenlet events associated with a given stack now.
Implements blueprint cancel-update-stack (partial)
Change-Id: Ic929d42946cf28eeb2a7caea8bf908f492693c09
When an error occurs during a stack create or update operation, don't
immediately stop the operation. Instead, wait up to 4 minutes for resoures
already in progress to complete before forcibly cancelling them. This
minimises the number of resources that will need to be replaced when using
the features from the blueprint update-failure-recovery to recover after
such a failed operation.
Change-Id: Ib08f3d8faea31f251a46fc0537983127f8ba2783
Replace the use of Resource.parsed_template() in updates with a frozen
version of a ResourceDefinition that uses the property values stored in the
database from the last update/create action, rather than calculated from
the template.
Change-Id: Id23ad60fbf98f2740ece2a337e795cfb3959d6f8
Implements: partial-blueprint update-failure-recovery
If the stack-update is switching from pre-HOT to HOT then
the update will fail because the functions are resolved
using the pre-HOT template.
This change uses the new template for the reparse instead of the
old one. It fixes the issue observed but any other effects are
unknown.
Change-Id: I2f749375ccd7ca917387ef96856b1fc248c8567b
Closes-Bug: #1356097
According to https://wiki.openstack.org/wiki/Python3 dict.iteritems()
should be replaced with six.iteritems(dict).
Change-Id: I9e2881a006433c8b44a3caccebc73bae4973a041
Use add_resource() and remove_resource() functions instead of indexing to
add and remove resources, to make it less surprising that the resources
themselves are being modified in the process.
Change-Id: I392cca8f68165988842affd1e23bce958645921b
Initialise resources with a ResourceDefinition object rather than a JSON
snippet from a template. We no longer need to store the json_snippet (since
the resolving of intrinsic functions is reversible when necessary), so get
rid of it.
Change-Id: I23d183d986ab1d771407502ec926e736d1f48d79
Commit https://review.openstack.org/#/c/94255/1 introduces check for
translating log messages. This check uses in pattern name 'LOG', so in
heat code also should be used this name intead of 'logger'. Also same
name is used in other projects.
Change-Id: Iba24c7eb1c13e68a91d090d7bcdbcb40d6e16071
Partial-Bug: #1321283
According to the OpenStack translation policy available at
https://wiki.openstack.org/wiki/LoggingStandards debug messages
should not be translated. Like mentioned in several changes in
Nova by garyk this is to help prioritize log translation.
Change-Id: I770dfc2fd474123a8ccc37311ef69d0c03e85545
Closes-Bug: #1318713
Previously the 'after' template passed to Resource.update() was completely
resolved. This led to an odd situation where the contents of a resource's
template snippet were either unresolved (but parsed) or resolved depending
on whether the resource had been updated or not. Resolve (ha!) this
discrepancy by passing a version of the template with the intrinsic
functions unresolved.
Change-Id: I8c11cdc3c28661db689b416833bde34cea12d98a
This patch is one in a series to re-enable H306 style check rule
(imports are in alphabetical order).
It touches engine files (without resources).
Implements: blueprint reduce-flake8-ignored-rules (partial)
Change-Id: Ifd36fc49199c48af891bd8e7cc2d2262a53032eb
Zuul