The abstract base classes previously defined in 'collections' were moved
to 'collections.abc' in 3.3. The aliases will be removed in 3.10.
Preempt this change now with a simple find-replace:
$ ag -l 'collections.($TYPES)' | \
xargs sed -i 's/\(collections\)\.\($TYPES\)/\1.abc.\2/g'
Where $TYPES is the list of moved ABCs from [1].
[1] https://docs.python.org/3/library/collections.abc.html
Change-Id: Ia282479bb1d466bd2189ebb21b51d91e89b9581e
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
A simple matter of having log lines fit
with all the others. It will now match the
ambiguous standard.
Also change comments inline to alias with log fixes.
Change-Id: I4a2ed6134f70c2965811f75ccb6ab8221fa0e960
Story: #2007319
Task: 38830
Six is in use to help us to keep support for python 2.7.
Since the ussuri cycle we decide to remove the python 2.7
support so we can go ahead and also remove six usage from
the python code.
Review process and help
-----------------------
Removing six introduce a lot of changes and an huge amount of modified files
To simplify reviews we decided to split changes into several patches to avoid
painful reviews and avoid mistakes.
To review this patch you can use the six documentation [1] to obtain help and
understand choices.
Additional informations
-----------------------
Changes related to 'six.b(data)' [2]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
six.b [2] encode the given datas in latin-1 in python3 so I did the same
things in this patch.
Latin-1 is equal to iso-8859-1 [3].
This encoding is the default encoding [4] of certain descriptive HTTP
headers.
I suggest to keep latin-1 for the moment and to move to another encoding
in a follow-up patch if needed to move to most powerful encoding (utf8).
HTML4 support utf8 charset and utf8 is the default charset for HTML5 [5].
Note that this commit message is autogenerated and not necesserly contains
changes related to 'six.b'
[1] https://six.readthedocs.io/
[2] https://six.readthedocs.io/#six.b
[3] https://docs.python.org/3/library/codecs.html#standard-encodings
[4] https://www.w3schools.com/charsets/ref_html_8859.asp
[5] https://www.w3schools.com/html/html_charset.asp
Patch 4 of a serie of 28 patches
Change-Id: I871c2dad10abc35790e730c7c4c5272f499b7623
If a stack has no tags, return an empty list [] instead of None for its
tags list.
Change-Id: I3c3c083d7261172d08b933de7c63b6a489ed6ccd
Depends-On: https://review.opendev.org/687769
Task: 37009
When updating a stack using the PATCH method (i.e. passing --existing on
the command line), the existing tags are kept unless otherwise
specified. However, while it was possible to set a new set of tags on
such an update, it was not possible to remove all of the tags because an
empty string was treated the same as None or a missing tags key (both of
which are treated as no change).
When the user explicitly requests an empty tags list (by passing ''),
honour that request.
Note that while None is treated the same as a missing key, in practice
the API never sends us a value of None; the key is never sent to the
engine if the user passes null to the API. The input string is split by
the API so the engine receives data with the key then the value is
always a Python list.
Change-Id: I6ae355b0a8af017c7a9145c9483c1ad3f2ac5ca5
Task: 37010
Avoid loading the tags from the DB and then re-saving every time
the stack is stored when the stack has tags. Avoid attempting to
lazy-load the tags from the DB multiple times when there are no tags.
Avoid lazy-loading the existing tags on an update when we intend to
overwrite them anyway. Avoid writing the same set of tags multiple
times.
e.g. in a legacy update, previously we rewrote the current set of tags
when changing the state to IN_PROGRESS, then wrote the new set of tags
regardless of whether they had changed, then wrote the new set of tags
again when the update completed. In a convergence update we also did
three writes but in a different order, so that the new tags were written
every time. With this change we write the new set of tags only once.
This could also have prevented stacks with tags from being updated from
legacy to convergence, because the (unchanged) tag list would get
rewritten from inside a DB transaction. This is not expected so the
stack_tags_set() API does not pass subtransactions=True when creating a
transaction, which would cause a DB error.
Change-Id: Ia52818cfc9479d5fa6e3b236988694f47998acda
Task: 37001
TripleO uses stack preview for deploy preparation tasks.
When there is a error in SoftwareConfig template, 'config'
results in null, without any debug or error message. Adding
log message to identify such errors easily.
Change-Id: I5700e091336fa11c7edcdd575ee27f0a49639ff8
In some scenarios, it is required to categorize the parameter.
Associating tags with the parameter, it is will be easy to
categorize the parameter and add simple validations to users.
This patch introduces a new attribute 'tags' with the parameters,
which is a list of strings.
Change-Id: I20fc95d606b0b8a08d3b46bf33f4860bff49c74f
Adding a if condition to format_validate_parameter function, is
failing the pep validation, because of complexity is going beyond
the 20 limit. Separating the constraints from the base makes it,
simpler.
Change-Id: Iaa4cdfceea67a17af1e17512cc28eb54dd52907d
Users can not view outputs of stacks in DELETE_FAILED.
But it's useful for magnum to get cluster info from stack.outputs
and for users to debug failed stacks.
Change-Id: Ib3927c61660b1b2162d5b34809b48401062a6f02
Close-Bug: #1712274
Add converge parameter for stack update API and RPC call,
that allow triggering observe on reality. This will be
triggered by API call with converge argument (with True
or False value) within. This flag also works for resources
within nested stack.
Implements bp get-reality-for-resources
Change-Id: I151b575b714dcc9a5971a1573c126152ecd7ea93
We will handle `show_deleted` param extraction in list stack API
call. Also we don't actually require to do extract it in extract_args.
Since extract_args is used in create/update stack which won't
require to have a show_deleted args.
Change-Id: I6b96dc1bb780e7658d7ef50c42fca81f12388551
Currently, we do live resolution of attribute values when in
list_stack_resources with detail enabled. However, in future we will cache
in the database those values that are referenced elsewhere in the template.
Since it's unlikely that every attribute would be referenced in the
template, this could lead to the API returning an inconsistent mix of live
and cached data.
Caching all of the attribute values on the off chance that someone might
request them through this obscure API would be expensive, and there are
also users that need to be able to access live resource attributes via
Heat. Therefore, always return live data for the resource attributes in
this API.
Change-Id: Ifea71b30299076d14d9bb91f3bce4dd7fc4c88a1
Related-Bug: #1660831
pydoc is part of the standard library and is much more robust at formatting
docstings than any trivial hand-rolled munger could be. For example, it
correctly preserves indentation within the docstring. Use it when
generating descriptions from docstrings for the API.
Change-Id: Ib565a64d990a45c652a9475255c37805f86070b4
Add `attributes_schema` and `_resolve_attribute` for OS::Keystone::User.
This patch allow get_attr works with OS::Keystone::User resource.
Partial-Bug: #1648004
Closes-bug: #1659923
Co-Authored-By: Rico Lin <rico.lin.guanyu@gmail.com>
Change-Id: I4e8e984c42120e23887ad9997df2fa530709be52
The list_events rpc api no longer includes resource properties data
when returning multiple events. In the "show event" case where an event
uuid is included in the list_events request, the resource properties data
for that single event continues to be included in the output.
Note that there is no change in behaviour on the heat-api side (see
heat/api/openstack/v1/events.py). Previously when listing multiple
events, it had just been ignoring the resource properties data it
fetched from the list_events rpc api, not including it in the output
it returned (e.g., to python-heatclient).
Change-Id: I7ac83d848cdd0e6c313870c0a4d59a5d9b2301f5
Partial-Bug: #1665506
This change adds project information of software configs
if using admin context.
Change-Id: Ia26919aa1177a9366c65710becb2097b79e02445
Closes-Bug: #1646312
When iterating through nested stacks in stop_traversal, there is no
need to load or process templates.
Change-Id: If2795cff4a9e7052e2186c811cdcd3e9451f9ff6
A modulo constraint will be used to restrict numeric values to
leave a certain remainder when divided with certain divisor.
E.g. we can use this to constrain values to even/odd numbers.
Change-Id: I9d7db4307be2a2b93cc928cf5912af7b49c72076
This is the last remaining area of the code where we were slinging around
CloudFormation-formatted snippets of template. Replace it with a proper API
along the lines of ResourceDefinition.
The default implementation of the new template.Template.outputs() method is
equivalent to what we have previously done spread throughout the code, so
that third-party Template plugins will not be affected until they have had
time to write their own custom implementations.
Change-Id: Ib65dad6db55ae5dafab473bebba67e841ca9a984
When oslo_utils isotime was deprecated, we switch to isoformat and lost
the timezone information. This switches back to a function which put the
Z suffix indicating UTC. It's a potential backward incompatible change,
but presumably better than sending incomplete information.
Change-Id: I78ab2dd025f1d00d4afaf72070e481aa8e09c5e8
Closes-Bug: #1607562
Adds 'description', 'tags' and 'updated_at' for stack
notification for projects such as Searchlight to store
stack metadata.
Change-Id: I7067e6649661d88cb86b4021a27ebbb6393cc7af
Closes-Bug: #1603163
These fields are lazy loaded, so cause extra unnecessary database
access.
The watch rules are needed when formatting every watch data metric for
the cloudwatch API, so an id->name map of all rules is built to avoid
repeated lazy loading of every rule.
Change-Id: I36dd43172ccd402f08a45b254ae47e208b7b8c08
Related-Bug: #1479723
The changes including:
1. Avoid hard code of resource and output keys
2. Decouple hot and cfn for outputs
Change-Id: I1fd7e08ff5c699ddfcf98c81aed5f0d91c4248b3
We support to get the deleted stacks by filter in stack-list
api, support to show the deleted stack by stack-id,
so to show the 'deletion_time' info of stacks.
Change-Id: I8c55fe7f6a899ca66aa3fef15c93195c03c5aa11
Closes-Bug: #1587214
Heat makes a lot of variations to the basic oslo.context which are going
to make it very hard to reuse with features added to the base
oslo.context.
There are a number of changes here that will make the heat context
options more like those from oslo.context.
*) context.user and context.tenant are IDs, not names. This will be
important for policy credentials.
*) kwargs should be passed through to base context so it can be extended
in the base class.
Change-Id: Ib0d60c6af196ba5c00459110b7a6190cff916d6f
The GET call to list events will support a nested_depth parameter. The
response will have an additional links url with the ref 'root_stack'
to indicate that this API supports nested_depth queries.
This has the following consequences for old/new combinations of
client/server
- new heatclient, new server - nested_depth param is set, server
returns nested events
- new heatclient, old server - nested_depth param is set, server
returns events with no root_stack, heatclient falls back to
recursive event fetch
- old heatclient, new server - nested_depth param is never set,
recursive event fetch works as before
Here are some timings for a TripleO overcloud stack with ~700 events.
Current heat and python-heatclient master:
time openstack stack event list --nested-depth 4 overcloud |wc -l
744
real 0m17.500s
This change, with heatclient 31278ff5f77b152b5ef7a4197e15c441c72ff163:
time openstack stack event list --nested-depth 4 overcloud |wc -l
608
real 0m1.725s
The difference in event count (744 vs 608) is due to the stack events
being filtered out for stacks with zero resources - these are a source
of unnecessary noise so their removal should be considered an improvement.
Closes-Bug: #1588561
Change-Id: I27e1ffb770e00a7f929c081b2a505e2007f5d584
This change removes the event.Event.load method, and switches
api.format_event to using the Event versioned object instead.
Since the only purpose of loading the stack is to get its identifier,
this is now also coming from a Stack versioned object.
Related-Bug: #1588561
Change-Id: I7b1fc99894818b44cde08cf08b010b1d1fb94e9f
In Python3, dict.keys() returns a view object rather than a list. This
behaves differently in that changes to the dict also modify the view, and
in that the view type interacts with various operators in different ways to
lists.
One universally correct transformation to preserve Python2 behaviour in
Python3 would be to replace all instances of d.keys() with
list(six.iterkeys(d)), and indeed we did. However, like many automatic
transformations the results are usually unsightly, invariably inefficient,
and frequently absurd. Not least because list(d.keys()) and indeed list(d)
are also equivalent.
This patch changes to using the simplest correct method of accessing the
data we want in each case.
This reverts or rewrites most of commit
4ace95ad47.
Change-Id: Iba3cf48246d8cbc958d8fb577cd700a218b0bebf
Since Iab7a997c0b4d9dcb8ceb3d2f49692216fe611df1 StackResource.has_nested()
returns True unconditionally. However, some parts of the code expect that
if has_nested() returns True, the caller will be able to call nested() and
not get None returned (this was the original definition of has_nested()).
This change restores the definition without the need to load the nested
stack (which is expensive, and the reason for changing this in the first
place).
This also allows the change from the original patch to be a bit tidier,
without a recurrence of the issue it initially caused (bug 1586906) that
was later fixed by I2ededcc57057f43206922b99bb72c43332336814.
Change-Id: Id26dc62b7513853b3f02a656518c59b3ff8d509a
Related-Bug: #1578854
StackResource is already building HeatIdentifier for the nested stack,
so this should be used by the resource formatter to avoid the overhead
of loading the nested stack.
Change-Id: Iab7a997c0b4d9dcb8ceb3d2f49692216fe611df1
Partial-Bug: #1578854
Currently calls to get a collection of stacks via the stack object
have the following data access behaviour:
- One query to fetch the stack records
- One query per stack to fetch the raw template
- One query per stack to fetch the tags
This causes excessive database round trips when there are many stacks.
In addition, the list_stacks call results in a collection of full
stack objects to be created which builds a fully parsed stack - most
of this information is then discarded by the RPC and middleware
formatters so this overhead is for no benefit.
This change is the first step in changing the data access patterns by
querying the Stack versioned object directly instead of via the full
Stack object. A future change will apply an eager fetch and caching
approach to avoid unnecessary queries.
This change does the following:
- Service list_stacks calls stack_object.Stack.get_all directly
- Service list_stacks formats with new function
api.format_stack_db_object with the exact fields required by the CFN
and REST API list stacks formatters
- Since the description field is the only one that requires full stack
parsing, it is now set to an empty string for stack listings via API
or calls to aws cloudformation list-stacks [1]
This last point may be controversial, my attempts to find uses of the
stack description in stack listings found none that do, and the following
API uses which *don't* show the description:
- heat stack-list
- openstack stack list
- horizon stack list
- rackspace control panel stack list
If we want to add the description back later we could always add a
description field back to the Stack table to store the denormalised
description.
[1] http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-describing-stacks.html
Partial-Bug: #1578851
Change-Id: I88b6f705e87ee7ff4acb7830dcddfc188ae6b3b2
The usage of resource.nova_instance property name is removed
and resource.physical_resource_id is used instead, except in
db schema. This patch takes care of this case, for the bug
fix made for resource filtering earlier.
Change-Id: Ic3aa5248232033fde0897a1b6400fa182e1c766a
Closes-bug: #1568974
We can search resources using this filter keys:
type, status, name, action, id, physical_resource_id. For more
information please see heat-stack-resource-search spec in heat-specs
repository and this commit: Iaae88f3b32bc2ba7b41a4078ef3aa8ffc07079b7
We pass `filters` as an argument to sqlalchemy that expects
filters keys to be a columns in a table. There are not such
columns `physical_resource_id`, `type` in Resource table, so
we get error while trying to filter resource by this keys.
Use `nova_instance` instead of `physical_resource_id` and filter
by `type` manually.
Closes-Bug: #1568974
Change-Id: I7b47eae000b244f58a05405746f5fcf9e922aff8
Don't filter out stack outputs when stack status is in progress,
returning it all the time instead.
Change-Id: Ife8f1eafe85e9ff4ce5f0a4692c64343c1c2d0b4
Closes-Bug: #1537693
For some reason we were using constants from a different data structure
(the resource schema) as field names in the resource data structure, even
though the names made no sense, just because they happened to have the
value we wanted. This change defines properly-named constants and puts them
in the RES_KEYS tuple where they belong.
Change-Id: Ia47ba64d85f0f4b93ba242b6d54e7f318fac7481
Stephen Finucane