Commit Graph

254 Commits

Author SHA1 Message Date
Takashi Kajinami 181d09d05f Replace deprecated import of ABCs from collections
ABCs in collections should be imported from collections.abc and direct
import from collections is deprecated since Python 3.3.

Change-Id: Ifb983fa478120a95760bf0cb78994210acdbe8e8
2021-08-02 08:27:12 +00:00
Erik Olof Gunnar Andersson e5d9d1b74f Use common rpc pattern for all services
This patch introduces a common rpc pattern to ensure
that the rpc transport is shared where possible. This
helps prevent rpc connection leaks and should ensure
that we are making the best possible use of all
available rpc connections.

Change-Id: Ib42e368cfda2b148a07df0bd74046739f40f7018
2021-01-19 12:24:08 -08:00
zhurong aa5f945088 Remove six murano/engine
Depends-On: https://review.opendev.org/#/c/720860/
Change-Id: Ib10d9e191ec93e55a0982d29de9a62a7b5d1894f
2020-04-18 01:08:08 -07:00
Andreas Jaeger d1dae85ac7 Update hacking for Python3
The repo is Python 3 now, so update hacking to version 3.0 which
supports Python 3.

Fix problems found.

Update local hacking checks for new flake8.

Change-Id: Ibe28b6b6bf3927e80816f0c5f1cb972fb6e2374b
2020-03-30 11:59:26 +02:00
zhurong 81acfe0f17 Add ip_version for NeutronNetwork subnet
Change-Id: I84791c93319dbd70fde436cc538b0e776e2a2a6d
2019-08-22 00:44:12 +00:00
zhurong 4ee97f7b7a Bump to hacking 1.1.0
Change-Id: I4019f41d9ddd843872855e6e80c0667ee2df758a
2019-07-04 11:51:47 +00:00
zhurong 95f9da9cd8 Fix pep8 F821 error
Change-Id: Idef9c90b57b67fae2e896de00cc8f647c238cae8
2019-01-09 08:37:25 +00:00
zhurong c023d63562 use get_rpc_transport to obtain an RPC transport instance
Change-Id: I4aaef5eb52500f597fa8270c71dd4816167e69cb
2019-01-08 19:43:11 +08:00
zhurong 9fe87010f5 Fix py37 tests failed due to async
Change-Id: If09cc4b27da22924e9ad5a5b930ccb0abcc84a34
Closes-Bug: #1786523
2018-08-11 15:53:06 +08:00
zhurong ed7941c346 Sanitizer some sensitive logs information
Murano engine maybe outputs some sensitive logs information,
This patch sanitizer the sensitive info.

Change-Id: Icd24a0388274454af335ce217614606b9bad0344
Closes-Bug: #1655265
2018-07-27 11:07:32 +08:00
Zuul 044a190e81 Merge "The original env should be {}, otherwise it can not be dealt with by heat client." 2018-07-19 06:17:53 +00:00
zhangyifan e47050853d The original env should be {}, otherwise it can not be dealt with by heat client.
Change-Id: If6013085d61ac87e836cea15fc47588753eabc1a
2018-06-19 14:26:45 +08:00
Ellen Batbouta 80b4e029bd Method, decrypt_data, uses internal Barbican endpoint
The yaql function, decrypt_data, in the file, yaql_functions.py,
sets the config option, barbican_endpoint_type, to internal.
This allows Murano to work when TLS is enabled.

Change-Id: Id9d82d7bbedcef87bdc9865ec55511d4e1f65507
Closes-Bug: #1775035
2018-06-04 14:18:12 -04:00
Stan Lagun 6f6897c132 Murano-engine side implementation of agent message signing
Change-Id: I1a23d185ac19f10c98d66f29a6930dfd17793954
Partial-Blueprint: message-signing
2018-01-15 16:56:54 +00:00
Guoqiang Ding 10c58ddce7 Fix the deprecated usage of "get_transport"
As log says, 'oslo_messaging.transport.get_transport()' is deprecated.
The reference link of oslo_messaging is at [1].

[1] https://review.openstack.org/#/c/454194/

Change-Id: I89061bd348988f9555f6bb77875bfdbf0aa76d07
2017-12-19 03:24:29 +00:00
Margarita Shakhova 3877f610ef Implement environment audit reports
Add notifications about environment events that are required for
tracking. These are AMQP notifications and oslo.messaging library
is used for sending them.

The follow event types are provided:

 - environment.deploy.end
   This event is issued on successful finish of environment deployment,
   provides general information about environment and also deployment
   start and finish times

 - environment.delete.end
   This event is issued on environment delete (and abandon as well)

 - environment.exists
   This is a period event, it's issued for every existing environment
   that has successful deployments

There are 2 new configuration options controlling these notifications:

 - stats.env_audit_period
   Controls how often to send environment.exists notification, by
   it's once per hour (60 minutes)

 - env_audit_enabled
   Allows to completely disable environment-related notifications. By
   default notifications are enabled.

Change-Id: I8dee2456b7ccab7c0c167aa21abb9710959ebb30
2017-08-24 10:11:35 +00:00
Jenkins 85bd657277 Merge "Add decryptData yaql function to murano engine" 2017-07-31 05:53:19 +00:00
Gyorgy Szombathelyi 78abdaa130 Use consistent session options
Various components defines different options for the client sessions.
Standardize them with the help of keystonauth1 lib.

Change-Id: I2f791caaf230a58b8426d1c1d6e1eb4316a85a28
2017-07-28 14:55:45 +02:00
Paul Bourke 9248605e67 Add decryptData yaql function to murano engine
Adds a new yaql function 'decryptData' which pairs with 'encryptData' on
the dashboard side.

Requires a valid secret storage backend (e.g. Barbican) to be configured
via Castellan in murano.conf, e.g.

    [key_manager]
    auth_type = keystone_password
    auth_url = <auth_url>
    username = <username>
    password = <password>
    project_id = <project_id>
    user_domain_name = <user_domain_name>

Murano will still work fine without this config but the encrypt/decrypt
functions will be unavailable.

Partially-Implements blueprint: allow-encrypting-of-muranopl-properties
Depends-On: I1be3a1e11e3f4c2170062927ad359bf679eb25d9
Change-Id: I09416b6d35ed2dafa823eca98262a4e23081e6eb
2017-07-21 13:32:13 +01:00
Jenkins ea4a304398 Merge "Replace six.iteritems() with .items()" 2017-05-03 20:35:15 +00:00
M V P Nitesh 580677eedc Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using six.iteritems to achieve
iterators. We can use dict.items instead, as it will return iterators
in PY3 as well. And dict.items/keys will more readable.
2.In py2, the performance about list should be negligible, see the
link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Change-Id: I45fa65427318e1c35bb521de46e81ea12ca7b770
2017-05-03 09:17:18 +00:00
Sam Pilla 3efd12c1c6 Add `nosec` for Bandit issue 506 in resource_manager.py
Running `tox -e bandit` will raise a `B506: Use of unsafe yaml load`
issue. Because yaml.safe_load is a wrapper for yaml.load(SafeLoader),
this is a non-issue raised by the tests. This patch adds a `nosec` to
ignore the issue and comments to explain why it is okay as is.

Change-Id: I4bb3b1635000a8bf77015f35f0be36df2c4f731f
2017-04-17 13:32:16 +00:00
shihanzhang 06746baa32 Remove log translations
Log messages are no longer being translated. This removes all use of
the _LE, _LI, and _LW translation markers to simplify logging and to
avoid confusion with new contributions.

See:
http://lists.openstack.org/pipermail/openstack-i18n/2016-November/002574.html
http://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html

Change-Id: I24ca32d9e92cdd2b3a91916337839d116e335084
2017-04-03 12:11:20 +08:00
Stan Lagun 78ffcfbab4 Agent initialization was fixed
When Murano Agent is disabled Agent was not
initialized properly which caused deployment errors
even if no agent calls made

Change-Id: I047b512cafe71650d6a5c7edc7863f0231292a22
Closes-Bug: #1670511
2017-03-09 15:39:29 +03:00
zhurong 4cbf1a27fe [Fix gate] Update oslo.messaging constructors following recent change
This patch fixes the murano gate to align with a recent change [0]
made to oslo.messaging in which the kwarg topic was changed to
topics.

[0] https://review.openstack.org/#/c/433950/

Change-Id: I9c70448914063976264dc40b10f238105bdfccc1
Co-Authored-By: zhurong <aaronzhu1121@gmail.com>
2017-03-07 20:19:38 +00:00
Stan Lagun c657ffce4d v1 execution plan template processing was fixed
In v1 execution plans Scripts section must contain a list
of base64-encoded files. But instead of a file content,
the file path was encoded.

Change-Id: Ibb45de6dd2895a308000d083e0603020502d3b0d
Closes-Bug: #1657839
2017-01-19 13:25:49 -08:00
zhurong 2eb086769f Correct the wrong calling 'getExternalNetworkIdForRouter'
In 'io.murano.system.NetworkExplorer', there is a
'get_external_network_id_for_network' method calling
'getExternalNetworkIdForRouter' to get external network id for specific
router interface, which should be 'get_external_network_id_for_router'

Change-Id: Id2046106e274b8f6682703659ab8e27233d49209
Closes-Bug: #1638923
2017-01-06 10:35:47 +08:00
LiuNanke 2506030cf1 Replace six iteration methods with standard ones
1.As mentioned in [1], we should avoid using six.iterXXX
  to achieve iterators. We can use dict.XXX instead, as it
  will return iterators in PY3 as well.
2.In py2, the performance about list should be negligible,
  see the link [2].

 [1] https://wiki.openstack.org/wiki/Python3
 [2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Change-Id: If90a56fad941e4bb55da1e9f14a8133983efc027
2016-12-01 16:07:09 +08:00
Stan Lagun 53abad42ab Always declare agent RabbitMQ queues
Murano uses 1 RMQ queue per agent + 1 queue for agent
listener. Listener queue is declared upon listener startup
which happens when the first agent command sent.
The agent queues were created by explicit call to Agent.prepare()
which was done from Instance.deploy() method.
This creates a problem if RabbitMQ server is restarted after deployment,
all non-persistent queues disappear and then an action get called. If
that action is not calling Instance.deploy() (because it assumes instance to be
deployed by that time) and sends a command to an agent it will never be
received.

This commit removes explicit prepare() method. Now the queues are going
to be created on demand, similar to listener queues do.

Change-Id: I21ae1efd483f32bb7c8e3dc1849af656b3bed56f
Closes-Bug: #1645419
2016-11-28 20:29:09 +00:00
Alexander Tivelkov 81eebd12ad Ability to retrieve current/owner user/project
Added an ability to retrieve information about the current user,
current project, environment owner (both user and project)
from keystone. Appropriate information (including
extra fields but excluding internal system data) is fetched from
Keystone using the same service credentials that are used to validate
tokens, create trusts etc.

- io.murano.User and io.murano.Project classes were added.
- Both classes have 2 static methods to get current and environment
   owner object of appropriate class
- Object model now contains project_id/user_id of the user who
   created the environment
- Deployment task contains project_id (renamed from tenant_id)
   and user_id of the user who initiated the deployment

Change-Id: Ic7e24c1d2b669ed315851047bcdb27e075cfc56b
2016-11-24 02:07:34 -08:00
Stan Lagun a19a66707b HeatStack async mode fix
Improve asynchronous push mode of HeatStack:
- Use spawn_after instead of spawn_after_local. Otherwise the data is never
   pushed if the initiated thread were to exit
- Cancel background thread instead of killing it. Cancel cancels the thread
   only if it hasn't started yet instead of killing it somewhere in the middle.
- Add post-execution cleanup to guarantee that async data push happens
   before the execution session end
- Make Instance destruction use async push to speed up the destruction
   in case when there are many servers and to test the HeatStack async mode

Closes-Bug: #1643702
Change-Id: I11d157844cb1d973d2cac62c2e6d67d047f75164
2016-11-22 16:02:12 +00:00
Jenkins 37cceebb76 Merge "Use method ensure_tree from oslo.utils" 2016-11-15 04:41:55 +00:00
LiuNanke af7d1cdedb Use method ensure_tree from oslo.utils
we don't have to def ensure_tree, because Oslo.utils 1.8
provides the function and just use it.I also deleted the
code in tests/unit/.

Change-Id: I61e9d87e6dde5b7a52e7b6399f4be6e8d5fca340
2016-11-14 19:45:48 +08:00
huoliang 1001243d85 Replace oslo_utils.timeutils.isotime
'oslo_utils.timeutils.isotime()' is deprecated in version '1.6' and
will be removed in a future version: use datetime.isoformat() instead.

Change-Id: I7ac6508510bc0fa131368c27dd8fdcababc37d0b
Closes-Bug: #1514331
2016-11-10 13:51:23 +08:00
Jenkins 69e4ab31ad Merge "Fix TypeError being throw by wait_ready in Engine System Agent." 2016-10-21 06:07:11 +00:00
Alexander Tivelkov 254202b98e Unblocking the gate
Skipped test_deploy_app_with_volume_creation_from_backup test

Tempest integration test is failing for volume creation from backup
(reported as bug #1634876). To unblock the gate the appropriate test
is skipped for now while the bug is being investigated.

AgentListener unit test didn't properly override configuration, so in
some circumstances it could be executed with the disabled murano agent
which was leading to errors.

At the same time AgentListener itself contained some legacy code
leading to under-initialized state in case if the listener was
disabled in config file.

Related-bug: #1634876
Change-Id: I5642b8017c61f77afb68463ba8eb3906eeae7225
Closes-bug: #1634483
2016-10-19 15:54:31 +03:00
Felipe Monteiro 1e5a1cbf80 Fix TypeError being throw by wait_ready in Engine System Agent.
Currently, wait_ready wrongly does call(template, False, timeout)
but the second param is expected to be a ResourceManager, leading
to the TypeError.

This fix replaces the wrong function call with
call_raw(template, timeout).

Change-Id: I57e92eae95efc3e68b51ea751d08dd87946f85cb
Closes-Bug: #1633176
2016-10-14 04:54:15 +00:00
Felipe Monteiro 922a6fac2e Fix TypeError being thrown by six.reraise in Engine Package Loader.
Currently, six.reraise throws a TypeError in Python 3.4, because
it is being called incorrectly in function load_package in
package_loader.ApiPackageLoader.

The way six.reraise is called was changed to reflect the
"generic" way it is called throughout the rest of the file.

Closes-Bug: #1632107

Change-Id: Iadb9c5a6483bf779afd916ebe53852b72581b03e
2016-10-10 17:32:37 -04:00
Jenkins 0fa1b7a006 Merge "Murano bindings to Glance Metadef API" 2016-10-04 16:32:02 +00:00
Jenkins 4020e0bea4 Merge "Fix typo and make docstring more clear" 2016-10-04 16:31:56 +00:00
Duong Ha-Quang 8d09fd5820 Fix typo and make docstring more clear
Rewrite docstrings in murano/common/wsgi.py to make it
more clear. Fix typo in other files.

TrivialFix

Change-Id: Iaa6bde5f3b0356b6ac9e67f39ebf0f7feff4ee45
2016-10-04 15:29:16 +07:00
Snihyr Kostyantyn 100d1edb39 Async HeatStack::push
This patch adds new feature - async HeatStack::push
If HeatStack::push is called with flag async=True,
template will be pushed in separate greenthread with
1 sec delay. If push operation is not finished and next
push is called, thread will be killed and next push
will be scheduled.

Change-Id: I8aea5a88fdf964b8ed0436f7d692dac50caf854b
2016-10-03 19:19:19 +03:00
Alexander Tivelkov 343f8b49e3 Murano bindings to Glance Metadef API
Added a python-backed class io.murano.system.MetadefBrowser providing
a read-only access to Glance Metadefinition Catalog. The class can
fetch namespaces mapped to a particular resource type and the contents
(objects) of a given namespace. Since the catalog is not updated
frequently the class provides caching capabilities for the fetched
data.

The MuranoPL part of the class provides a single method called
canBeAppliedTo which may be used to check whether some tag can be
applied to a particular type of resource.

Since Glance connectivity is required, a new configuration section is
added to provide capabilities to tune conection properties (url,
endpoint type, encryption etc).

Change-Id: I8ea70e314dbe701b462fb1620376965af717c10d
Targets-blueprint: meta-attribute-assignment-and-propagation
2016-09-28 20:14:38 +03:00
Boden R 6fd6c097ac Replace retrying with tenacity
We are replacing all usages of the 'retrying' package with
'tenacity' as the author of retrying is not actively maintaining
the project. Tenacity is a fork of retrying, but has improved the
interface and extensibility (see [1] for more details). Our end
goal here is removing the retrying package from our requirements.

Tenacity provides the same functionality as retrying, but has the
following major differences to account for:
- tenacity uses seconds rather than ms as retrying did.
- tenacity has different kwargs for the decorator and
Retrying class itself.
- tenacity has a different approach for retrying args by
using classes for its stop/wait/retry kwargs.
- By default tenacity raises a RetryError if a retried callable
times out; retrying raises the last exception from the callable.
Tenacity provides backwards compatibility here by offering
the 'reraise' kwarg.
- tenacity defines 'time.sleep' as a default value for a kwarg.
That said consumers who need to mock patch time.sleep
need to account for this via mocking of time.sleep before
tenacity is imported.

This patch updates all usages of retrying with tenacity.
Unit tests will be added where applicable.

Note: This change is not newton critical so projects are welcome
to hold off on committing until post-newton. Ideally this change
will merge by the first part of Ocata so dependant functionality
can land and have time to solidify for Ocata.

[1] https://github.com/jd/tenacity

Change-Id: I18a0075e9a7c376b6881555ce67267a8944caafa
2016-09-22 10:56:34 -06:00
Jenkins 835a6a4f18 Merge "[messaging] Using get_notification_transport()" 2016-09-19 23:33:58 +00:00
Jenkins 62f1a0fcf2 Merge "Prevent logging of result of resources.string() method call" 2016-09-14 20:19:57 +00:00
Valerii Kovalchuk ce8e2ec37a Prevent logging of result of resources.string() method call
The result of resources.string() method is the content of the file
saved to string. Logging of binary file content causes
UnicodeDecode error. Moreover, logging of any file content can be
security issue. Finally, it is just not practical to fill the logs
with tons of text from the big files.

Change-Id: I87077b002f2a8888c22b4dfba1f7b9f0508fec69
Closes-bug: #1561522
2016-09-14 17:44:56 +00:00
Stan Lagun 2b57eb3fca Serialization of destruction dependencies
If the object was deleted through the API between deployments
the only way for those who used to subscribe to its destruction
to be notifyed upon next deployment is to persist destruction
dependencies to the object model.

This commit adds such serialization and deserialization.

Also move GC class from engine to dsl

Targets-blueprint: dependency-driven-resource-deallocation
Closes-Bug: #1619248
Change-Id: Icd2e882be5770244aa1ecafe265aff1439ebec9e
2016-09-14 03:48:00 +00:00
Stan Lagun dc050d41cb GC.isDoomed() and GC.isDestroyed() functions were added
io.murano.system.GC.isDoomed() static method was added. It can be used
within the .destroy method to test if other object is also going to be
destroyed.

io.murano.system.GC.isDestroyed() static method was added. It checks if
the object is destroyed and thus no methods can be invoked on it.

Targets-blueprint: dependency-driven-resource-deallocation
Change-Id: I9ca51f342be27e88e149f217b41145becdfbe232
2016-09-14 03:47:19 +00:00
Stan Lagun dcbf5e1899 Raise exception on call of method of destroyed object
In theory using destruction dependencies one can obtain a new
reference to an object that is about to be destroyed. In this
case any attempt to call a method on such object should result
in exception being raised. This guarantees that the nothing can
be done on the object after its .destroy method was executed.

Also because the HeatStack instance is owned by the CloudRegion
it is going to be destroyed prior to it. But CloudRegion needs
to call $stack.delete() in its .destroy method and this will cause
error. As a solution HeatStack made not to be owned by the
region and instead be an independent object with destruction
dependency on it which guarantees that it will be destroyed
after the CloudRegion.

Targets-blueprint: dependency-driven-resource-deallocation
Change-Id: I5650ea672d5f121be69669f27dd5f513fbdd5c44
2016-09-14 03:44:36 +00:00