The abstract base classes previously defined in 'collections' were moved
to 'collections.abc' in 3.3. The aliases will be removed in 3.10.
Preempt this change now with a simple find-replace:
$ ag -l 'collections.($TYPES)' | \
xargs sed -i 's/\(collections\)\.\($TYPES\)/\1.abc.\2/g'
Where $TYPES is the list of moved ABCs from [1].
[1] https://docs.python.org/3/library/collections.abc.html
Change-Id: Ia282479bb1d466bd2189ebb21b51d91e89b9581e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 4 of a serie of 28 patches
Change-Id: I871c2dad10abc35790e730c7c4c5272f499b7623
Previously, in 1fff951639, we added outputs
to the generated templates of various nested stack resources that
referenced any attributes required to calculate the parent resource's
attributes. That was necessary because attribute values are not cached
unless they are referenced somewhere. It isn't necessary for reference IDs
(i.e. the results of get_resource) because those are always available.
However, providing reference IDs as outputs as well will allow us to get
all attributes of the parent resource from outputs of the nested stack.
Unlike ResourceGroup/Chain, in InstanceGroup and AutoscalingGroup the order
and content of the member in list-type outputs cannot be determined at
template generation time (failed resources are excluded, and the sort order
depends on creation time). Therefore, always use dicts instead of lists as
outputs, both for the new reference ID outputs in AutoscalingGroup and
existing attribute outputs. For ResourceGroup/Chain, create both dicts and
lists as appropriate, so that each attribute value can be obtained directly
from an output.
Change-Id: I3139246a5acb17df9018e7c7a2cf88f740b3f8be
Partial-Bug: #1731349
When we add a resource to a template using add_resource(), strip any
references to resources that are not present in the template from
'depends_on'. This means that resources that are present and depended on
but conditionally disabled in the source template will not be depended
on in the destination template, where they're not present at all (and
thus would cause errors when the template is used).
Change-Id: I58a64cd523cc0d1da8cd39793f0c839189458d99
Closes-Bug: #1701677
Related-Bug: #1649900
This helps with dynamically building templates, and is similar to the
add_resource method.
Change-Id: I51ce5c913438083d6d43eb6d8a8bf55cfa8ea3ee
Related-Bug: #1660831
Previously we implemented the str_replace intrinsic function by repeatedly
calling str.replace(), with the result that some text could be subject to
multiple substitutions (i.e. some new text that was inserted by a
replacement could itself be partly replaced if some part of it matched
another key), although users could not rely on this happening since the
replacement order is explicitly undefined.
We anticipated that users would guard against problems here using some sort
of guard characters like %replace_me% (although we avoided requiring any
particular choice of characters). Over time it has become clear that for a
variety of reasons (of varying legitimacy), this is often not the case.
Sometimes the format of the source data is not under the direct control of
the user.
This patch revises the algorithm to perform the replacements in a single
pass and makes the ordering of keys deterministic. It also refactors the
various versions of the function so that we no longer need two
implementations of the core algorithm.
Change-Id: I43461a0d78226f921018efb32aae6e2255928bd0
This change supports:
1. Allow boolean value or condition function as
condition name of 'if' function:
resources:
r1:
...
properties:
a: {if: [true, 'value_true', 'value_false']}
r2:
...
properties:
b: {if: [{equals: [...]}, 'value_true', 'value_false']}
2. Allow boolean value or condition function as
condtiion name in resource definition:
resources:
r1:
...
condition: false
r2:
...
condition: {and: [cd1, cd2]}
3. Allow boolean value or condition function as
condition name in outputs:
outputs:
output1:
value: ...
condition: true
output2:
value: ...
condition: {not: cd3}
Change-Id: I2bf9bb0b608788c928d12425cbfdaf658df9e880
Co-Authored-By: Zane Bitter <zbitter@redhat.com>
Blueprint: support-conditions-function
Parse each field in a resource definition individually instead of all in
one go. This allows the HOT and cfn formats to apply different parsing to
different fields - specifically in the future we want to parse the
condition field like a condition, using only the valid condition functions.
Change-Id: I5b864d241a5e16c09fcce30c40d634d9bb72e173
HOT is our canonical native format, so define the cfn intrinsic functions
in terms of the HOT functions instead of the other way around. Ensure each
function has a consistently-formatted docstring appropriate to its template
type.
Change-Id: I743446bdef3fd7472537db75bf9e0fbfb164e36a
Change the definitions of the Template.conditions() method to return a new
heat.engine.conditions.Conditions object. This class encapsulates the error
handling involved in looking up a condition, and lazily caches resolved
values so that conditions we never reference are not evaluated.
By raising ValueError instead of StackValidationFailed inside
Conditions.is_enabled(), we ensure that the user sees the full path to the
erroneous condition name if it appears inside the 'if' intrinsic function.
Change-Id: I266eec7e87d185182c4c0fea864e7ebdafa11bd5
Now that we raise them in only a single place each, there's not a lot of
value in having a custom exception type for condition-related errors in the
template any more, and in fact there was no code to handle them
appropriately in heat-api so it would have resulted in 500 errors to the
client if they ever escaped heat-engine anyway.
Just use the existing StackValidationFailed exception type. This provides
more flexible control over the path than InvalidConditionReference did.
Also, raising ValueError in the Invalid function means that
StackValidationFailed will be raised with the path, so it will be much
easier for users to find where they have used an invalid function.
Change-Id: Iaa6ca030fc144efc75e707910f84bfa7385fa9e2
Override the default Template.outputs() method (which is designed to mimic
how previous versions of Heat interacted with the Template) with a
Cfn/HOT-specific one, and mark the default implementation as deprecated.
This also allows us to considerably simplify the filtering out of outputs
based on conditions. Support for conditionals is removed from the default
implementation of Template.outputs(), since it didn't appear in any
previous release of Heat and therefore isn't needed for backwards
compatibility. (Third-party template format authors who want to implement
conditionals in the future - and those who don't - should override the
outputs() method.)
Change-Id: Iecf1ec9e0c467ec70ee9beeb9b02daccc8b8082c
Closes-Bug: #1618713
This reverts commit 4a8ad39a38.
There was no protection against circular references causing infinite
recursion.
Change-Id: I35226420c458474e75716591d587e73a82e75688
Closes-Bug: #1621529
This change allows reference with other conditions by name in
definition of a condition, something like:
conditions:
cd1: {equals: [{get_param: env_type}, 'prod']}
cd2: {not: cd1}
cd3: {equals: [{get_param: zone}, 'fujian']}
cd4: {and: [cd1, cd3]}
Change-Id: I4fbc3839977d2d4299d65d4d56e3d41d56d07aef
Blueprint: support-conditions-function
This allows us to parse the conditions only once instead of reparsing them
all every time we encountered a resource with a condition. It also allows
us to get rid of some of the API surface area and reduces our reliance on
some of the rest.
Partial-Bug: #1618713
Change-Id: I3c2273722171b9c4cb13ef6588f7b522b6689b1c
We have to hard-code knowledge of how to translate from template data to
ResourceDefinition objects, and we know what data we need in any of the
Template implementations to do so. So just get rid of the extra
indirection, non-locality of data, doubling up of definitions of valid
keys, and assumptions about the relationship between keys and
ResourceDefinition arguments and do everything directly for the condition
and external_id fields like we do for all of the other arguments.
Change-Id: I8e6381ed8ae7ffe4711653e473ab1412e4476be9
This method was added in order to be able to validate the resource
definitions during the Stack.validate() phase, and not have to wait until
the Resource objects are created. (It can't be done during
Template.validate() because the stack is not passed there, which is needed
to call Template.parse().)
However we actually already generate the ResourceDefinitions right below
this in Stack.validate() anyway, by virtue of creating the Resource
objects, so there was really never any reason to put it in a separate
method. It was actually much less efficient because we had to parse the
resource section of the template twice instead of once.
Accordingly, move the validation code into the resource_definitions()
methods and deprecate the validate_resource_definitions() method. (We have
to keep calling it for now, because third-party template formats may be
relying on this validation having been done before calling
resource_definitions().)
Change-Id: I76be34c5cf8500fca1654a066a5cead57b0e7bab
This is a utility function for validating the types of the various fields
in a resource definition during the validation phase, prior to creating the
ResourceDefinition object, that was added in
3f16eebd06.
Not only does this not belong in the public interface for all third-party
Template classes, but the implementation is nonsense. The verification code
must call it with a hard-coded key from the list of allowed keys along with
the allowed types for that value, and the method checks that the key is in
the list of allowed keys. (Surprise! It always is.) Meanwhile the keys
passed by the user are not checked for invalid values.
Fortunately the validity of the data passed by the user is checked
elsewhere as a side-effect of converting the resource definitions from HOT
to cfn, so we haven't been allowing bogus HOT templates since Kilo.
In the cfn-style templates we have never checked for invalid keys in the
resource definitions (and thus we can't really start), so the current
implementation was accidentally correct.
Since this is part of the public third-party interface we can't fix it
immediately. Instead, deprecate it and override it with a sane
implementation in the common parent class of HOT and CFN templates that was
added to hold such things. We'll remove it entirely from the public
interface later.
Change-Id: Ie38e8c62c5303f72a43f24fe1ace7ce8696abde1
We're supposed to be phasing out any code that still treats templates like
JSON data in favour of using the Template APIs. Since conditions are
already accessed only through the API, we should enforce that by adding it
to the list of keys that cannot be accessed directly in HOT and CFN.
Change-Id: Iea58c3d0fcc15f4834688c187319f40dc8691357
Ensure that any Template implementation will get a sensible default value
for _parser_condition_functions (which is used in the root Template class's
concrete implementation). Most implementations will need only to provide
condition_functions in order to get the desired behaviour; others such as
HOT can continue to override it in their own __init__ methods.
Change-Id: I05078277c056b6c204d484e69b92eb93252feb92
Support 'not' and 'Fn::Not' for templates:
AWSTemplateFormatVersion.2010-09-09
heat_template_version.2016-10-14
Change-Id: I6a9c89a23160a2cf06c37677871bcfbfab9599be
Blueprint: support-conditions-function
Provides output condition for cfn/hot templates,
if the condition of output evaluates to false, will
set output value to None.
Change-Id: I0398e39541a4176ef5699331c10536c59f1cb3e7
Blueprint: support-conditions-function
Purpose is to avoid str_replace from doing partial replace
when a shorter instance of a matching key is found.
Change-Id: I63c13f4eb3b1375cb2df1a34fd4f09561564f400
Co-Authored-By: Zane Bitter <zbitter@redhat.com>
Closes-Bug: 1600209
New an intermediate class named CommonTempalte to put the commont
implementation of HOT and CFN templates.
This patch move the resource definition validating to the
new class.
Change-Id: I96ce57d64f0649e309e40e75e573371bde917fbe
Blueprint: support-conditions-function
This changes:
1. Support 'Conditions' for AWSTemplateFormatVersion: 2010-09-09
2. Support 'conditions' for heat_template_version: 2016-10-14
3. There is no need to add a new HeatTemplateFormatVersion template,
because we decide to support conditions in AWSTemplateFormatVersion
and heat_template_version: 2016-10-14, so remove the
HeatTemplateFormatVersion.2016-10-14
4. Move the definition of function 'equals' to hot/functions.py
5. Mark 'equals' as condition function which supported in conditions.
Change-Id: I2e7bdfa1c2052e75f35f2bd0003cdc170188d8b8
Blueprint: support-conditions-function
Now error with a path to the problem part of template will be
raised while parsing template. It affects both resource
definitions and outputs parsing. Note, that during parsing of
template we only instantiate the function objects, but doesn't
make any function validation. The function validation still
doesn't return any path with error. This will be fixed in the
other patch.
Change-Id: Ic5b960e79a54e88087803bc092f614911d7e995a
This will make it possible to find resource keys without having to
reference a specific template format module.
Change-Id: I1a2b3e36ecd20100c316ba9b2ce19aabe9c9ef70
The changes including:
1. Avoid hard code of resource and output keys
2. Decouple hot and cfn for outputs
Change-Id: I1fd7e08ff5c699ddfcf98c81aed5f0d91c4248b3
In Python3, dict.keys() returns a view object rather than a list. This
behaves differently in that changes to the dict also modify the view, and
in that the view type interacts with various operators in different ways to
lists.
One universally correct transformation to preserve Python2 behaviour in
Python3 would be to replace all instances of d.keys() with
list(six.iterkeys(d)), and indeed we did. However, like many automatic
transformations the results are usually unsightly, invariably inefficient,
and frequently absurd. Not least because list(d.keys()) and indeed list(d)
are also equivalent.
This patch changes to using the simplest correct method of accessing the
data we want in each case.
This reverts or rewrites most of commit
4ace95ad47.
Change-Id: Iba3cf48246d8cbc958d8fb577cd700a218b0bebf
Provides condition function 'equals' for hot template,
'Fn::Equals' for HeatTemplate, which versions
are 2016-10-14.
Change-Id: Ib0ffa76e6c562dfbddca5f9dce807f2c6ea3eb82
Blueprint: support-conditions-function
Don't treat a ResourceDefinition as a snippet of CloudFormation template in
the resource_facade intrinsic function.
This is slightly tricky, because in the case of the update_policy we don't
necessarily have a schema and we just want to pass the (resolved) data
verbatim, so we have to access a non-public attribute of the object.
Change-Id: Ib23a4be4f0a250a292e0ec707b1d5e885f15ed7e
Make sure to create stack successful if 'deletion_policy'
section using function 'get_param':
1. Allow Function as valid type when 'deletion_policy'
section validation.
2. Resolve 'deletion_policy' data before deletion_policy
checking when rsrc_defn init.
Change-Id: Idee0ed90c34cfbfb09546969b144496d757195d8
Closes-Bug: #1556790
This partially reverts https://review.openstack.org/#/c/275602/, which
removed the validation of the parameters argument completely, but a
comment added post-merge rightly points out that we can just check for
a map or function and still fix the bug.
Also, it seems that we had an existing test which failed to catch the
fact that this didn't work, so fix that to prove the expected
behavior works in the context of a stack, not just when resolving a
function snippet, and add a test which proves the reinstated
constructor validation works as expected.
Change-Id: I21c1868061e37760e1cb27f44ebb7c4ee97fefef
Partial-Bug: #1539737
Currently a validation error occurs if you attempt to reference
via get_param or get_attr when using str_replace, however if
we avoid the explicit validation in the constructor it actually
works fine, and we already have a validation to handle any
invalid input elsewhere in the function definition.
Change-Id: Ib934f443a8b8e4f75335a9d8b992e7f86791aa45
Closes-Bug: #1539737
Resource plugins will now override rsrc.get_attribute method to add
extra logic to Fn::GetAtt specific to resource implementation.
Also provides proper comment to rsrc.get_reference_id method.
Change-Id: I7561afa1c824b2294dd6701be6a19723abbe0522
Closes-Bug: #1531840
Fn::Select try to cast its index argument to integer in all the cases,
but it also needs to use it as a string when you give it a mapping. This
moves the cast after the mapping case has been handled.
Change-Id: I84ff404d3a26fb51c2048e43851b1ef5e3031d67
Closes-Bug: #1518337
Heat supports 'list_join' function to join one or more
lists of strings in Liberty release. But when we resolve
the function, we pop some arguments, this introduced
errors if resolve the snippet again when updating a resource.
This patch correct the behavior of 'list_join' function,
also refactor some codes for the functions of Kilo/Liberty.
Change-Id: Ia67a0171261ca1a864fe11f7d61c3bf5ef801040
Closes-Bug: #1513060
Don't check None attribute name if want to get all
attributes of resource, and this only works in template
'2015-10-15' or higher version.
Change-Id: I6699bfc8bc8096e5dc313d01b3a829694c105fea
Closes-Bug: #1505560
To get attribute instead of return None if resource
is in SNAPSHOT or CHECK action.
This patch allow to get_attr for hot template version
'2013-05-23' also.
Change-Id: I318f4c158ebe0a4afbb3f0bcbe9d856d90f0a4bb
Closes-Bug: #1505054
Stephen Finucane