To assist with automated configuration validation, we need entry points
for oslo.config.opts for yaql and cache options.
Change-Id: I228282d2c05f6583fe972470b326e7182f635b39
Templates can be complex and contain multiple instances of
map_merges. Debugging this can be complex, so this change
aims to improve the mapping and help users determine where
the problem has occured.
With this change, instead of just saying an error occured,
we will now return the object that the error occurred with
and a note about it's type. This will allow for quick
identification of what the problem was.
Change-Id: I1c4da88029bbf1a3cfb58f5e21da28ec2912f924
The abstract base classes previously defined in 'collections' were moved
to 'collections.abc' in 3.3. The aliases will be removed in 3.10.
Preempt this change now with a simple find-replace:
$ ag -l 'collections.($TYPES)' | \
xargs sed -i 's/\(collections\)\.\($TYPES\)/\1.abc.\2/g'
Where $TYPES is the list of moved ABCs from [1].
[1] https://docs.python.org/3/library/collections.abc.html
Change-Id: Ia282479bb1d466bd2189ebb21b51d91e89b9581e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 6 of a serie of 28 patches
Change-Id: Ic56cf6da5ff9ab8f4bd218e6516b87f47165206d
list_concat checks that all args are lists. If that fails, the exception
will now show which args aren't a list, and show all the args. This is
useful context to help find the error in the template.
Change-Id: Ic88257fd36bdde884e8ac4f17186c42b76ec538b
In a number of places we parse templates passing None as the stack
definition, in order to be able to extract information from the template
in cases where we don't actually need to resolve functions.
However, the GetParam and GetFile HOT functions require an actual stack
definition to be passed in order to initialise them. So far we haven't
encountered any cases where those functions appear where we pass None
for the stack definition. Ensure that all Functions can always be
created even with no stack definition.
Note that the cfn ParamRef class is not affected, as the Ref function
always selects the GetResource implementation when the stack definition
is None.
Change-Id: Ic947dcdc17e9dbc58459a92ba970c292d70578cd
* Add Rocky code name
* Add new HOT version for Rocky release
The new version is "2018-08-31" or "rocky".
* Add sem-ver flag
Sem-Ver: api-break
Change-Id: I261b6c28b8b7ee9e75ca9a895155a656ef82cd0d
The patch https://review.openstack.org/#/c/527001/ bypassed the None
list but did not consider the case where all the for_each lists are
getting their values from attributes and therefore are None.
This patch set fix it by also skipping the check if all the values
are None -- as len(value_lens) would be 0 in that case.
Closes-bug: #1732934
Change-Id: Iba574bbd3877f9ca1ca755c384b3ddb47aec711c
Attributes may return "None" before the resource is created,
so the length check for for_each lists in repeat function will fail.
Bybass the length check for the None list
Change-Id: Ida81060e62340b3ea37ba0fdd7443dc88ba6c5aa
Closes-bug: #1732934
In some scenarios, it is required to categorize the parameter.
Associating tags with the parameter, it is will be easy to
categorize the parameter and add simple validations to users.
This patch introduces a new attribute 'tags' with the parameters,
which is a list of strings.
Change-Id: I20fc95d606b0b8a08d3b46bf33f4860bff49c74f
When we add a resource to a template using add_resource(), strip any
references to resources that are not present in the template from
'depends_on'. This means that resources that are present and depended on
but conditionally disabled in the source template will not be depended
on in the destination template, where they're not present at all (and
thus would cause errors when the template is used).
Change-Id: I58a64cd523cc0d1da8cd39793f0c839189458d99
Closes-Bug: #1701677
Related-Bug: #1649900
Override the default implementation of all_dep_attrs() (which calls
dep_attrs() in a loop). This makes all_dep_attrs() for GetAtt and its
arguments as efficient as dep_attrs(), even if we only need the data for a
single resource (as in convergence after we update a resource).
Change-Id: Ia6a4d132e655ef24fb158959d05967d7f1c7fef8
Related-Bug: #1684272
This unifies the 'lightweight stack' used in convergence with the how
things work the rest of the time: we now always obtain resource data from
the StackDefinition's ResourceProxy objects. This means that when we are
checking an individual resource, we will never create all of the other
Resource objects for the stack in memory (although we already avoided
loading data for them from the database) - now we will only create
ResourceProxy objects as needed.
Change-Id: Id7472557e26d172df88841ff7f20afdd7f5bfada
Implements: blueprint stack-definition
This check was added in 0638030671 because
originally the Resource objects other than the one we were checking were
not initialised with any data during a convergence resource check.
Therefore they would be in the INIT_COMPLETE state, but would return valid
attribute values from the cache data.
It was made redundant by 1956ddd2a6, which
loaded the resource state in the Resource objects. The state will never be
INIT_COMPLETE.
This code requires the Function's 'stack' member to be a Stack object, so
we have to get rid of it in order to be able to start using a
StackDefinition object in its place.
Change-Id: I1a97401bf126a3ec10cc420bbb0786b785035d48
Adds a new section 'permutations' for 'repeat'
function to decide whether to do nested loops
to iterate over all the permutations of
the elements in the given lists, the default value
is True if no specify and keep the same behavior as
before.
For example:
repeat:
template:
network: %net%
port: %port%
ip: %ip%
for_each:
%net%: [n1, n2]
%port%: [p1, p2]
%ip%: [ip1, ip2]
permutations: False
Will be resolve to:
[{network: n1, port: p1, ip: ip1},
{network: n2, port: p2, ip: ip2}]
Change-Id: I2a008400fb71453f6a78656f2e041ae2efa098a2
Blueprint: improve-repeat-function
Ensure that in convergence we can calculate the dep_attrs of a resource,
including the attributes referenced in outputs, even if:
- One of the resource's attributes is referenced in an ouput; and
- The attribute path itself also contains a get_attr function; and
- The attribute referenced by the inner get_attr function in the output is
not referenced by any resources.
Since the attributes referenced only in outputs (and not in resources) are
not passed to dependent resources in the NodeData, the data required to
resolve the inner get_attr function is not available. Therefore getting the
dep_attrs for the relevant resource from the output definition would fail.
The next patch does so during the calculation of the NodeData, and any
resulting exception is not caught (bug 1703043), so the stack would end up
stuck IN_PROGRESS.
To avoid this, ignore any errors resolving attribute path components in the
course of calculating the dep_attrs of a given resource. This may mean that
an attribute value required for the outputs is not stored in the database,
but it can still be resolved live later.
Change-Id: I2ed1e3dacd371e36656559ba92dfada05df0ceba
Related-Bug: #1660831
Correct function name in description for test function.
Also correct function sample in hot function class `MapMerge`.
Change-Id: I61acec48e4a8365940d3e6e3d0a93d0c58fbda43
This helps with dynamically building templates, and is similar to the
add_resource method.
Change-Id: I51ce5c913438083d6d43eb6d8a8bf55cfa8ea3ee
Related-Bug: #1660831
Add new function 'contains' to check whether
the specific value is in a sequence. The resolve
result of the new function is a boolean value, so
we put it as a condition function too.
Change-Id: I90074070dd917df13345fb7c8d4ed09e4a76e864
Blueprint: support-contains-function
Now we support the map and list valuse of
the "for_each" argument to 'repeat' function,
but the error message tells "...must be lists",
it's inappropriate.
Closes-Bug: #1672936
Change-Id: I3bbe370772be7d01e63c1e88afffbd507ff75507
The current output can be a little confusing in the case where
you have multiple make_url calls in one template - adding the
args to the output gives you a better chance to debug as you
can e.g compare the path with the template.
Change-Id: I73a0ffa8ee7d6a7f7a210548c966eb9faf6f5f5b
Closes-Bug: #1693490
Adds function 'list_concat_unique' to concat
several lists using python's extend function
and make sure without repeating items.
Blueprint: support-list-concat-unique-function
Change-Id: Ia0e35eb578ebe91543c644155a86030b3258118b
Now the function 'list_concat' allows string type
args, like {'list_concat': 'I am string'} or
{'list_concat': ['v1', 'v2']}, this patch fixes the
validation to avoid string type args.
Change-Id: If6d73a90e25f38a37c0503fe0b1aaa93a6fbc380
Closes-Bug: #1685434
for n, f in six.iteritems(self.functions):
if not isinstance(f, hot_funcs.Removed):
self._parser_condition_functions[n] = function.Invalid
else:
self._parser_condition_functions[n] = f
In the above codes, "f" is a class ,not an instance, therefore isinstance is uncorrect.
Change-Id: I9ec41d19b77acf6e083b7c5cb5b779dc54b4e547
The already existing str_replace_strict function raises an error if a
param is not present in the template. str_replace_vstrict, newly added
in this patch, is identical but also raises an error if any of the
params have an empty value.
Change-Id: I5407135cc0435cfbad2d18964fe2119c999f67a3
This function uses python's extend function for lists in order to
concatenate lists. So, given the input: [['v1', 'v2'], ['v3', 'v4']]
the result would be ['v1', 'v2', 'v3', 'v4'].
This comes as an alternative to using yaql's concat.
Change-Id: I082833e73388540b29f71b78293fd36fe04b7b92
Query strings are not _required_ to have slashes escaped, and in fact RFC
3986 suggests that "it is sometimes better for usability to avoid
percent-escaping those characters" (Section 3.4).
TripleO is currently unable to use the make_url function because its
database URLs contain slashes in the query, and end up in config files that
don't allow percent characters. This is probably not going to be an
isolated occurrence, so don't escape slashes in the query string.
Change-Id: I72f84e737b042ecfaabf5639c6164d46a072b423
A large proportion of uses of the str_replace function is to build URLs out
of various components from various sources. This is invariably brittle,
with a failure to escape special characters, deal with IPv6 addresses, and
so on. The make_url function provides a both a tidier interface and correct
handling of these edge cases.
Change-Id: I61b6dff01cd509b3d1c54bca118632c276569f4e
To generate its NodeData, a Resource must know which of its attributes
other parts of the template have dependencies on, which it determines by
asking the intrinsic functions in the template. Currently, the get_attr
function when used with no arguments depends on all of the attributes of
the resource (except 'show'), and it obtains this list from the Resource
itself (via the attributes_schema). This presents a chicken-and-egg problem
if we want to give Functions access only to a ResourceProxy based on the
NodeData.
Resolve this by returning a constant sentinel value (indicating that all
attributes are required) to the Resource, which can determine for itself
what attributes it has.
Change-Id: If209b589278fe1facb06e157fe76209fdd42a8a2
Partially-Implements: blueprint stack-definition
When an intrinsic function serialises one of its arguments to JSON, sort
the keys in dicts so that the output is stable over time.
Change-Id: Ib509717837afff3b183707e2b4f99191fe572271
We should allow the case where the keys collide, but the
replacement is the same as the current key (e.g a noop).
This is useful when you want to do a map_replace based on a
user parameter, then the default can be to do a noop replace
but the user parameter may also be any other (non colliding)
value.
Change-Id: I8cc5761c219616b4f8c18b3f44c4bf864c5457f1
Closes-Bug: #1652034
David Vallee Delisle