In some cases, some resources may require more than
one service_extension. In this case, we should allow
resources to define a list of required resources.
Change-Id: Iee0104a741cc050047824d23b8ab5ee9871c4f28
Use retry_state parameter that contains all information about
current retry invocation in the prepare_attempt callback.
Change-Id: Ib47b6335b591850965132448befb7b3dbe31984a
Task: 44471
With I04e7ad90944c2d03ce0e59ba16af9d60d6e01222 we allowed
update of previously-existing resource if replacement
creation has failed during last update. However, we force
update the status of the existing resource to 'COMPLETE'.
Therefore, in the next update if properties/type has not
changed for the resource it won't try to update/replace
the previously existing resource resulting in false positive
of stack updated successfully.
Task: 42194
Change-Id: Icc90a921ec67e49aec2c4acfad72235c57c78421
The abstract base classes previously defined in 'collections' were moved
to 'collections.abc' in 3.3. The aliases will be removed in 3.10.
Preempt this change now with a simple find-replace:
$ ag -l 'collections.($TYPES)' | \
xargs sed -i 's/\(collections\)\.\($TYPES\)/\1.abc.\2/g'
Where $TYPES is the list of moved ABCs from [1].
[1] https://docs.python.org/3/library/collections.abc.html
Change-Id: Ia282479bb1d466bd2189ebb21b51d91e89b9581e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Now that we only support Python 3, use the 'yield from' keyword instead
of the @wrappertask decorator for calling co-routines from a co-routine.
This should greatly simplify backtraces when calling nested co-routines.
Change-Id: If9beaff74cf4facbc4aa4b30f31a3a087bdcad8a
This patch gives effect to the change intended to be implemented in
6bc25ef0e0.
The Resource._do_action() method is a co-routine, but in a number of
less common resource actions, we simply return the result of calling it
rather than turning the action into a wrappertask and yielding it. This
works fine (the action returns a generator; the caller doesn't care
where it came from) and reduces the amount of wrapper frames on the
stack.
However, in most cases we want the stored properties to be used while
calling _do_action(). However, to ensure that they are actually in place
when the co-routine is running, we must yield rather than return the
generator. In the suspend, resume, snapshot, and chekc actions we were
not doing that. It is not known whether any resource types are actually
accessing properties during any of these actions.
Change-Id: I2b2a87ccf1988821a250373fea261c1850e5c51b
The method Resource.check() is not a generator function, but it does
return a generator in most cases. Simply calling the function does not
cause the resource to actually be checked. Only in the case where the
resource did not provide a handle_check() method would it do anything at
all - and in that case it simply changes the state to CHECK_COMPLETE.
Yield the generator function so that the resource will actually be
checked if the resource type supports it.
Change-Id: I2a78b6f6235529a95838307e3d38731d31ff5c4b
Task: 37828
A simple matter of having log lines fit
with all the others. It will now match the
ambiguous standard.
Also change comments inline to alias with log fixes.
Change-Id: I4a2ed6134f70c2965811f75ccb6ab8221fa0e960
Story: #2007319
Task: 38830
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 7 of a serie of 28 patches
Change-Id: I7c50cb96b2991dcf843ac99199b31fb3c8a899fa
OS::Mistral::Workflow resource creates a mistral workflow with
a unique name (resource_id). We replace FAILED resources by
default and replace wont work in this case as it will try to
use the same workflow name for the replacement resouce, if the
'name' property is provided.
If the workflow does not exist/deleted using mistral api directly,
it would create a replacement resource, but it would delete the
workflow when cleaning up the old resource. So we would endup
with a replacement resource without any backing workflow.
This adds a new resource attribute ``always_replace_on_check_failed``
and overrides needs_replace_failed() for OS::Mistral::Workflow.
Task: 38855
Change-Id: Ia0812b88cae363dfa25ccd907ecbe8b86f5b1a23
If the parent resource of a nested stack is locked due to an IN_PROGRESS
update, cancel the nested stack update (which will result in the parent
resource being marked FAILED and releasing the lock so that the new
traversal can begin acting on it). This also cancels all descendants of
the nested stack.
This means that a concurrent update no longer gets blocked at a nested
stack boundary until the previous update has finished.
Change-Id: I5f14453ebab75d89672c6eea12de46d48a5147f3
Task: 17760
We don't seem to do it anywhere else. TripleO uses hook
reason string comparsion which does not work if
internationalized.
Story: #2005035
Task: 29524
Change-Id: I3e8f4587966969084cbf1e38406cd3cff23eb9a6
The convergence prototype ignored checks of resources that had previously
been replaced by another resource. However, it turns out that this was
unnecessary in the final version of the prototype:
e9d2f833a5
Keeping this limitation prevents us from allowing users to roll back to a
previously-existing resource if creating the replacement fails, other than
by using the automated rollback mechanism (which is special-cased).
Since the graph for the traversal is calculated at the start of the update,
there's no need to limit which nodes are allowed to be checked. Only the
selected resource will have an is_update=True node in the traversal graph.
Removing this additional restriction allows us to improve the algorithm
that chooses which resource to update. Since a new traversal ID is stored
before we calculate the traversal graph, we can guarantee that no new
resource checks will start (though some may still be happening) while the
algorithm is choosing, so there's no danger of it picking a resource that
later gets replaced.
Since we're no longer treating automatic rollback as a special case, reset
the replaced_by field whenever we update a previously-replaced resource.
Change-Id: I04e7ad90944c2d03ce0e59ba16af9d60d6e01222
Story: #2003579
Task: 26197
Use a single write to the database to convert each resource. Add a
method to the versioned object class that encapsulates the DB-specific
information, and get rid of the Resource.set_requires() classmethod that
just calls a method on the versioned object instance that's passed to
it.
Change-Id: Ieca7e0f0642c38c44fb8d7729333a0ccd93c9cb4
When a mock hasn't been provided with a long-enough list of side-effects
for the number of times it is called, it raises StopIteration (instead
of something sensible like AssertionError). Prior to Python 3.7, this
exception could bubble up until it just stopped a task. (In Python 3.7
it will eventually be caught and turned into a RuntimeError.)
To ensure that any errors of this sort are not handled silently, and to
enable us to test for them prior to using Python 3.7, catch any
StopIteration errors coming from plugin-provided non-generator functions
and convert them to RuntimeError exceptions.
This reveals many errors in the unit tests, many introduced in the
process of converting from mox to mock, which are also fixed by this
patch.
Change-Id: I5a1eff6b704dff7c17edcbbe58cdbc380ae6abc9
Story: #2003412
Task: 24553
We assume that only resources that are in INIT_COMPLETE need to be created
(as opposed to updated). If we run a check on a resource that hasn't been
created yet, we shouldn't change the status to CHECK_COMPLETE because
thereafter we will try to update it.
Changing the status to CHECK_FAILED (as we do if the resource supports
handle_check()) is OK because the resource will get replaced on the next
update anyway.
Change-Id: Iaadf3b57b0e8edc76d338c017dbd05bf2a37698f
Story: #2003062
Task: 23102
If there are resources in a template that don't exist in the database at
the time of a stack check (or suspend or resume) operation, running the
stack check will cause them to be stored in the database. Since these
operations have not been converted to convergence (story 1727142), they do
not set the current_template_id as a convergence update would. If this
occurs then the stack will be unrecoverable.
To avoid this, when convergence is enabled store any missing resources in
the same manner that a convergence update would, prior to running the stack
check/suspend/resume.
Just in case, make sure the stack doesn't get stuck if we do end up in the
wrong state, by not trying to load a template with None as an ID.
Change-Id: Iedba67c5de39dc2d58938da5505dda5dd147c130
Story: #2003062
Task: 23101
Sometimes when deleting resources, we use parent resources in api
calls. If the parent resource is deleted then the children
could be assumed to be deleted. So ignore those translations errors
and return unresolved values when accessing properties.
Change-Id: I2392b5c46920ac6da518fe33bdcba3e7a9fe17a8
Story: #1747836
Task: 23278
When translating name to id, if we don't find the resource it's
better to ignore the error and return the non-translated value
for before properties.
Change-Id: Ic25c35098cff8b68e184a336b21f2e8df70bb1ee
Story: #2003015
Task: 23103
If resource creation fails with an HTTP Conflict error, this is presumably
due to a race condition with one of the underlying services. In this case,
allow Heat to retry the resource creation so that hopefully in most cases
the user will never have to deal with the error.
Change-Id: I301bb28231fcdc249f86ec1a2f09cb993023785b
Story: #1745010
Task: 17344
We want to be able to call is_not_found() or is_conflict() from the client
plugin from the Resource class, regardless of what resource plugin is
implementing the resource type. This is only possible when the
default_client_name is set, which is optional for plugins to do. As a
result, when handling an exception we need to check explicitly that there
is a default_client name, or (as in Resource.validate_external()) we will
get an assertion error.
To simplify the avoidance of programming errors, define a
_default_client_plugin() method that *always* returns a plugin whose
is_not_found() and is_conflict() methods can be called.
Change-Id: I1005397187457d9be42ba86096644dd17287d0e7
Story: #1745024
Task: 17677
During the original prototype development, I missed the fact that the
`needed_by` field of a resource is no longer needed for the actual
convergence algorithm:
c74aac1f07
Since nothing is using this any more, we can avoid an unnecessary DB
write to every resource at the start of a stack update. For now, just
write an empty list to that field any time we are storing the resource.
In future, we can stop reading/writing the field altogether, and in a
subsequent release we could drop the column from the DB.
Change-Id: I0c9c77f395db1131b16e5bd9d579a092033400b1
In convergence, resources can continue to exist in the database that are
no longer part of the latest template. When calling
Resource.required_by() for such resources, we still want to get a list
of those resources that depend on them. Previously we did this using the
`needed_by` field in the resource. Since this is the only actual use of
needed_by, get the information from the Stack's convergence graph
instead (which is generated from the resources' `requires` field and
ignores `needed_by`). This eliminates any risk that the `requires` and
`needed_by` fields get out of sync, and allows us to get rid of
`needed_by` altogether in the future.
Change-Id: I64e1c66817151f39829d5c54b0a740c56ea8edad
We seem to leave replaced_by field of a resource after rollback
(if a stack cancel is issued before the replacement
resource is added to the stack). This in turn results in the
stack getting stuck in IN_PROGRESS for further updates.
Change-Id: I3361ad90e303d24a16a943408fe0f6c406cf785f
Co-Authored-By: Zane Bitter <zbitter@redhat.com>
Task: #19582
Story: #2001974
We would need old resource definitions when doing rolling update of certain
group resources like RG/ASG. Therefore update the file map with the files
from old template.
Change-Id: I8f880e5b23c25159ecab1c63b594329d8df33973
Closes-Bug: #1765454
Task: #17360
Story: #1765454
Since we are doing set operations on it, make the 'requires' attribute of a
Resource a set and only convert to/from a list when loading/storing. To
avoid churn in the database, sort the list when storing it.
Change-Id: I137fae8ae630eb235d7811fcba81738d828e6a1e
Takashi Kajinami