The abstract base classes previously defined in 'collections' were moved
to 'collections.abc' in 3.3. The aliases will be removed in 3.10.
Preempt this change now with a simple find-replace:
$ ag -l 'collections.($TYPES)' | \
xargs sed -i 's/\(collections\)\.\($TYPES\)/\1.abc.\2/g'
Where $TYPES is the list of moved ABCs from [1].
[1] https://docs.python.org/3/library/collections.abc.html
Change-Id: Ia282479bb1d466bd2189ebb21b51d91e89b9581e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 13 of a serie of 28 patches
Change-Id: I09aa3b7ddd93087c3f92c76c893c609cb9473842
The cfn Ref instrinsic function is implemented as a Python function
rather than a class, so calling issubclass() on it fails in Python 3.7.
Change-Id: I3f826cd8f024b591d7c3b41ad1e3f960558d3f9b
It was unclear what the valid arguments to Template.get_section_name() were
(especially since the function is mis-named for what it actually does in
HOT). Define the arguments as constants and don't pass string literals any
more. Be consistent in how we define paths, standardising on the method in
Resource.validate_template().
Change-Id: Ifd073d9889ff60502f78aaa54532cec2b7814d93
This helps with dynamically building templates, and is similar to the
add_resource method.
Change-Id: I51ce5c913438083d6d43eb6d8a8bf55cfa8ea3ee
Related-Bug: #1660831
To be able to handle parameter defaults properly, we need to load all
possible templates when merging the environments, to generate an
approximate schema.
Closes-Bug: #1669571
Change-Id: Idab781d3e6af92990c357a727aa60ec0aa8f5f05
Ensure that the caching we do of conditions in TemplateCommon always
applies to the stack that has been passed in as an argument.
This was not the case previously, and in fact it was required not to be the
case because in the convergence path the new parameter values were not
copied into the stack at the start of an update. This patch rectifies that.
Change-Id: I942b53570222bf4e0d1bf78e584792d445c14090
Change the definitions of the Template.conditions() method to return a new
heat.engine.conditions.Conditions object. This class encapsulates the error
handling involved in looking up a condition, and lazily caches resolved
values so that conditions we never reference are not evaluated.
By raising ValueError instead of StackValidationFailed inside
Conditions.is_enabled(), we ensure that the user sees the full path to the
erroneous condition name if it appears inside the 'if' intrinsic function.
Change-Id: I266eec7e87d185182c4c0fea864e7ebdafa11bd5
Now that we raise them in only a single place each, there's not a lot of
value in having a custom exception type for condition-related errors in the
template any more, and in fact there was no code to handle them
appropriately in heat-api so it would have resulted in 500 errors to the
client if they ever escaped heat-engine anyway.
Just use the existing StackValidationFailed exception type. This provides
more flexible control over the path than InvalidConditionReference did.
Also, raising ValueError in the Invalid function means that
StackValidationFailed will be raised with the path, so it will be much
easier for users to find where they have used an invalid function.
Change-Id: Iaa6ca030fc144efc75e707910f84bfa7385fa9e2
This reverts commit 36bf170f34.
The construction and validation phases of intrinsic functions are
deliberately separated, to:
- Reduce the amount of unnecessary work done
- Reduce the risk of changes to validation breaking the loading of existing
stacks from the database
- Ensure that the most critical errors are reported to the user first
This patch reverts a commit that sought to combine the two phases. There
was no linked bug and no explanation other than that this 'improved'
things.
It does not, however, restore the check for the Removed function during
properties translation (and the associated test), because I believe its
original existence was based on a misconception. There is no way in
practice for a hot.functions.Removed object to appear in the properties a
translation rule is operating on.
Change-Id: I59d02e540771baf4fe078081cd0993a3251eb9a6
Override the default Template.outputs() method (which is designed to mimic
how previous versions of Heat interacted with the Template) with a
Cfn/HOT-specific one, and mark the default implementation as deprecated.
This also allows us to considerably simplify the filtering out of outputs
based on conditions. Support for conditionals is removed from the default
implementation of Template.outputs(), since it didn't appear in any
previous release of Heat and therefore isn't needed for backwards
compatibility. (Third-party template format authors who want to implement
conditionals in the future - and those who don't - should override the
outputs() method.)
Change-Id: Iecf1ec9e0c467ec70ee9beeb9b02daccc8b8082c
Closes-Bug: #1618713
This is the last remaining area of the code where we were slinging around
CloudFormation-formatted snippets of template. Replace it with a proper API
along the lines of ResourceDefinition.
The default implementation of the new template.Template.outputs() method is
equivalent to what we have previously done spread throughout the code, so
that third-party Template plugins will not be affected until they have had
time to write their own custom implementations.
Change-Id: Ib65dad6db55ae5dafab473bebba67e841ca9a984
Following on from the deprecation of validate_resource_definitions() from
the Template interface, also remove validate_condition_definitions(). The
definitions will be validated when calling Template.conditions(), which
now occurs when loading the resource definitions, so it will still happen
during Stack.validate().
Since this interface is new and has never appeared in a release, we have no
need to go through a deprecation process.
Change-Id: I4dc36433c1aca776c2151205ed26a0f5241a750a
This method was added in order to be able to validate the resource
definitions during the Stack.validate() phase, and not have to wait until
the Resource objects are created. (It can't be done during
Template.validate() because the stack is not passed there, which is needed
to call Template.parse().)
However we actually already generate the ResourceDefinitions right below
this in Stack.validate() anyway, by virtue of creating the Resource
objects, so there was really never any reason to put it in a separate
method. It was actually much less efficient because we had to parse the
resource section of the template twice instead of once.
Accordingly, move the validation code into the resource_definitions()
methods and deprecate the validate_resource_definitions() method. (We have
to keep calling it for now, because third-party template formats may be
relying on this validation having been done before calling
resource_definitions().)
Change-Id: I76be34c5cf8500fca1654a066a5cead57b0e7bab
This is a utility function for validating the types of the various fields
in a resource definition during the validation phase, prior to creating the
ResourceDefinition object, that was added in
3f16eebd06.
Not only does this not belong in the public interface for all third-party
Template classes, but the implementation is nonsense. The verification code
must call it with a hard-coded key from the list of allowed keys along with
the allowed types for that value, and the method checks that the key is in
the list of allowed keys. (Surprise! It always is.) Meanwhile the keys
passed by the user are not checked for invalid values.
Fortunately the validity of the data passed by the user is checked
elsewhere as a side-effect of converting the resource definitions from HOT
to cfn, so we haven't been allowing bogus HOT templates since Kilo.
In the cfn-style templates we have never checked for invalid keys in the
resource definitions (and thus we can't really start), so the current
implementation was accidentally correct.
Since this is part of the public third-party interface we can't fix it
immediately. Instead, deprecate it and override it with a sane
implementation in the common parent class of HOT and CFN templates that was
added to hold such things. We'll remove it entirely from the public
interface later.
Change-Id: Ie38e8c62c5303f72a43f24fe1ace7ce8696abde1
Ensure that any Template implementation will get a sensible default value
for _parser_condition_functions (which is used in the root Template class's
concrete implementation). Most implementations will need only to provide
condition_functions in order to get the desired behaviour; others such as
HOT can continue to override it in their own __init__ methods.
Change-Id: I05078277c056b6c204d484e69b92eb93252feb92
A Macro differs from a function in that it modifies the syntax tree
directly, so that the result of inserting a macro in the tree might not be
just different output, but actually be based on resolving different
functions. A macro can remove syntax from the tree, modify it or fabricate
it from whole cloth prior to parsing.
The proposed If 'function' for implementing conditionals is a macro.
Arguably the Repeat function should be implemented as a macro also.
Change-Id: I2c5afe31d509ffedc9e0b2ef237f52d04687e0be
Partially-Implements: blueprint support-conditions-function
Provides output condition for cfn/hot templates,
if the condition of output evaluates to false, will
set output value to None.
Change-Id: I0398e39541a4176ef5699331c10536c59f1cb3e7
Blueprint: support-conditions-function
This changes:
1. Support 'Conditions' for AWSTemplateFormatVersion: 2010-09-09
2. Support 'conditions' for heat_template_version: 2016-10-14
3. There is no need to add a new HeatTemplateFormatVersion template,
because we decide to support conditions in AWSTemplateFormatVersion
and heat_template_version: 2016-10-14, so remove the
HeatTemplateFormatVersion.2016-10-14
4. Move the definition of function 'equals' to hot/functions.py
5. Mark 'equals' as condition function which supported in conditions.
Change-Id: I2e7bdfa1c2052e75f35f2bd0003cdc170188d8b8
Blueprint: support-conditions-function
Its impractical to pass in the current context to
TemplateFiles._refresh, and it will soon be mandatory for all db api
context arguments to be not None. This change creates a short-lived
context just for the refresh.
It will have the same effect as the current behaviour of creating a
new session, since the a new session will be associated with the
context, so there is no danger of returning stale data from the
current session's identity map.
This change also makes the context argument mandatory for any raw
templates operations that might need one.
Change-Id: I156096d736aac4999a3eebe077ed50ef7384ca02
Move validation code to __init__ method for these list of
functions: Removed, Repeat, Yaql, so now validation error will
be found during template.parse() and returned with a path
to the function in template. Don't change validation for
GetAttr, because it resolves some arguments and we need to parse
template before we can use resolve().
Also, remove check for Removed function in translation, because
now we can't have Removed in properties.
Change-Id: I4f2e1ebee37e82badcdca9110cadc649aa01be8c
Now error with a path to the problem part of template will be
raised while parsing template. It affects both resource
definitions and outputs parsing. Note, that during parsing of
template we only instantiate the function objects, but doesn't
make any function validation. The function validation still
doesn't return any path with error. This will be fixed in the
other patch.
Change-Id: Ic5b960e79a54e88087803bc092f614911d7e995a
Currently this is reflected in the user_env_as_dict output, which means
not only is it stored in the DB (which we want), but also passed around
to nested stack (which we don't want because it's derived from the parent
stack hidden parameters), and also to the user via stack environment show
(the API returns it but heatclient currently hides it), which we also don't
want because it's not a valid key in user-provided environments.
Change-Id: If5821ccb4a8bbf98012a2541ddd3c8e91455e5cc
Closes-Bug: #1590507
The changes including:
1. Avoid hard code of resource and output keys
2. Decouple hot and cfn for outputs
Change-Id: I1fd7e08ff5c699ddfcf98c81aed5f0d91c4248b3
There should always be a context supplied, lets use its session
directly so we can be sure that the session is the shared one from the
context.
Any remaining functions which call get_session() directly will be
handled in their own changes as some require special behaviour.
Change-Id: I3e52d2ff6bb1014f201715a1b1ba32c122c2391d
Related-Bug: #1479723
In Python3, dict.keys() returns a view object rather than a list. This
behaves differently in that changes to the dict also modify the view, and
in that the view type interacts with various operators in different ways to
lists.
One universally correct transformation to preserve Python2 behaviour in
Python3 would be to replace all instances of d.keys() with
list(six.iterkeys(d)), and indeed we did. However, like many automatic
transformations the results are usually unsightly, invariably inefficient,
and frequently absurd. Not least because list(d.keys()) and indeed list(d)
are also equivalent.
This patch changes to using the simplest correct method of accessing the
data we want in each case.
This reverts or rewrites most of commit
4ace95ad47.
Change-Id: Iba3cf48246d8cbc958d8fb577cd700a218b0bebf
Save space in the db by allowing templates (especially nested
templates) to reference the same template files dictionary objects,
now stored in the table raw_template_files. Also, cache template files
per heat-engine process as a RAM/performance optimization.
Also, begin to allow for future rolling upgrades with
oslo.versionedobjects.
Change-Id: I1842ea6ba6773a135b1007b793a5f0884417747d
Closes-Bug: 1570983
In case of convergence, delete is an update of stack with empty
template. The issue with it is that the stack-show will not show the
parameters used to create/update the stack. The fix is to retain the
environment from the last template in the empty template and then update
the stack with that template so that the stack-show will show the
parameters properly.
Change-Id: Ia45ad5d024456cd583c282a42b26c9528512e73c
Closes-Bug: #1583673
The process of bringing us into compliance with the H405 pep8 rule has left
us with a lot of docstrings that either don't follow the recommendations of
PEP0257, are misleading, wrong, or nonsensical. This patch attempts to fix
them up.
Change-Id: Icfe90b5ba3102a7e13ab659038a8b2af9019e9e6
test_template
* sort list of versions in ascending order before returning it
to maintain consistency
* encode to utf-8 before hashing in test cases
test_template_format
* cast float to int explicitly
* convert offending character to bytes because pyyaml expects
bytes in it's ReaderError exception
partial blueprint heat-python34-support
Change-Id: Ic2f84e9db1f3f7229f40e4afbc62948d9e7be6ff
There are three tests skipped which needs further
addressing later on. However, fixing test_stack should prevent
future regressions for when a stack object is created.
partial blueprint heat-python34-support
Change-Id: I15bfd0990ebe248436c1a0c5e3f4f9a8332c2a57
As part of stack create and preview, template is being
validated twice, first at service._validate_new_stack()
and second at stack.validate() by self.t.validate().
This patch fixes this issue.
It also improves code readability by using
proper indentation.
NOTE: second part, will have dogpile cache backend
integration
Change-Id: I86649752cadcd145760d482d07313881183b02b0
Closes-bug: #1444316
Refactored the empty template and added it to template module so that it
can be re-used else where. In convergence it is needed for rolling back
when create fails. Also needed for deleting stack, that is, update the
stack with empty template.
The empty_template method can be passed a template header (version) in
order to get an empty template of that version, otherwise by default, an
empty HOT template is returned.
Change-Id: Ic59bd2d4b9cfed1efd1973af0c4ec9274bcc204a
The heat gate is broken due to other libraries pinning
on pbr<1.0
Stevedore extension loading currently has verify_requirements=True.
This change removes this to use the default of False. Loading extensions
is likely the wrong place to be enforcing requirements versions of
libraries that heat depends on, so lets not do it.
This is likely to also improve heat robustness for deployments with
inconsistent dependencies, not just this current heat breakage.
Change-Id: Ifee37e00412e824eead9ba1435ea5a328ff3df58
Closes-Bug: #1458399
Stephen Finucane