Commit Graph

77 Commits

Author SHA1 Message Date
Andy Botting 277db4ff4e Fix write mode for execution plan queue
This commit disables writing in binary mode for the execution plan.

The contents being written comes direct from json.dumps, which is
always returned as a string.

Change-Id: I60cab27f29bf1b538dcf348b100d200048d9fd25
2022-11-15 13:33:32 +11:00
Andy Botting 449c50d170 Remove use of anyjson module
The use of the anyjson module causes issues with Python 3.10 with
the following error:
`error in anyjson setup command: use_2to3 is invalid.`

The module itself hasn't been updated since 2012 and the source [0]
now errors with a 404, and there doesn't seem to be any real benefit
over the standard python json module.

[0] https://bitbucket.org/runeh/anyjson/

Change-Id: I00ae5e9545136c1b930f8cf4036b902a326e1c54
2022-10-12 13:55:45 +11:00
zhurong 89b4512e0d Fix lower-constraints job
Change-Id: I672d275490c9098f9f8835658921002416609da1
2021-04-07 09:15:02 +08:00
Andy Botting 7d481ad60e Fix another base64 python3 issue
This is to address another usage of base64 that was missed from
commit 794c47f3b8823e97a0f019064e4b74a5ee204d04

Change-Id: I9bbff8084d82928deeafa6a55c487ed0f665b470
2020-10-30 19:29:19 +11:00
Andy Botting 3c41b4f1b9 Fix base64 usage for Python 3
Python 3 now handles str and bytes differently to Python 2, so when
using b64encode/decode, the input and output must specifically by
bytes.

This commit adds some base64 helpers to coerce the input to become
bytes if the input is a str and decodes the output back to str after
the encode or decode.

It also adds a test case to replicate the issue and cover the new
functions.

Change-Id: I8f64da03c7cfc76e620cb3b98201f6287b752bb3
2020-10-08 18:50:22 +11:00
Hervé Beraud 08273d5c23 Use unittest.mock instead of mock
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.

Change-Id: I9d33976afc6b4d475f11f3f6b751ae7093cf2a04
2020-06-10 11:31:07 +02:00
Corey Bryant f2e41f2f65 Monkey patch original current_thread _active
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592

Change-Id: I0e8f80bb2e7acfb00e5ff47e0aa983afb7ee6e38
Closes-Bug: #1863021
2020-05-05 15:41:18 -04:00
Sean McGinnis ec54d366a3
Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: I24daffca6667325bd7710e02e0b6a057935e9ac9
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-04-18 11:55:15 -05:00
zhurong 84f5778fc5 Remove six usage
Change-Id: Ia6b08e81f8df43e67bed0fbd082d7d7edd178fbc
2020-04-16 04:34:44 -07:00
Andreas Jaeger 7d8894441a Update hacking for Python3
The repo is Python 3 now, so update hacking to version 2.0 which
supports Python 3.

Fix problems found.

Update local hacking checks for new flake8.

Change-Id: Ifa4b61c66561c114f6e35355de51f0e2d525b5d5
2020-03-30 08:24:41 +02:00
Chuck Short 407676319d Fix unit tests for python3.6
Make sure we are passing a string when we are running
the test_script.runner_downloadable otherwise it will fail
when using python3.6 complaining about the following:

TypeError: expected str, bytes or os.PathLike object, not MagicMock

Change-Id: Id2e4f3d6389e1855051f791d95103e7c5d449f49
Signed-off-by: Chuck Short <chucks@redhat.com>
2018-08-23 10:22:45 -04:00
yatin cab8de9496 Fix: config generation
Review https://review.openstack.org/#/c/530865 renamed
storage_opt to opts but missed to rename it in
muranoagent/opts.py. This patch fixes it.

Change-Id: I41f06279147f0d2aafe80efeb1c0e558f2698d5e
2018-01-19 14:06:51 +05:30
Stan Lagun 3341975bab Murano-agent side implementation of message signing
Change-Id: I50077cff89d5caed1d23bc4e5d7dfaa5dbae599e
Partial-Blueprint: message-signing
2018-01-03 02:14:26 -08:00
Stan Lagun 2468fb5939 Tighten access to runtime agent folders
Murano Agent uses default folder permissions for the
execution plans and scripts. If the default is too permissive
(which is unusual), other users on that machine can trick
the agent to execute malicious execution plans by putting files
into queue folder and use it to get the root privileges.
In most common sense users won't have write permissions to murano-agent
folders. However, they can hijack execution plans and other data
that might contain sensitive information.

This commit sets 0700 mode to the agent runtime folders so that they
can be accessed only by the user that runs the agent (+ the root,
if it's someone else).

Change-Id: I27f0495a509c4d1435d630e2bc5bfdf3549486d5
2017-11-22 10:00:26 -08:00
Gerry Buteau 7473dc7306 Mask sensitive information before logging execution plan.
Use oslo_utils.strutils to mask any potentially sensitive
information in the execution plan before writing to the log.

Change-Id: I9008dcd68da4ba14bbf1360a672e1a91ae0a8e91
Closes-Bug: #1706059
2017-07-24 10:50:23 -04:00
Ngo Quoc Cuong 860288305f Remove usage of parameter enforce_type
DeprecationWarning: Using the 'enforce_type' argument is deprecated in
version '4.0' and will be removed in version '5.0': The argument
enforce_type has changed its default value to True and then will be
removed completely."

Change-Id: I3e29f9c443f76fa3acaf0267a0c42618f89961d3
See: https://review.openstack.org/#/c/328692/
2017-06-27 21:27:26 -04:00
jeremy.zhang 1698fa0992 Replace uuid.uuid4() with uuidutils.generate_uuid()
Openstack common has a wrapper for generating uuids. We should
use that function to generate uuids for consistency.

Change-Id: I5d0e5523595088d934ee233c48e403f23326cee9
Closes-Bug: #1082248
2017-03-24 16:57:21 +08:00
jeremy.zhang 18758fab1a Use HostAddressOpt for opts that accept IP and hostnames
Some configuration options were accepting both IP addresses
and hostnames. Since there was no specific OSLO opt type to
support this, we were using ``StrOpt``. The change [1] that
added support for ``HostAddressOpt`` type was merged in Ocata
and became available for use with oslo version 3.22.

This patch changes the opt type of configuration options to use
this more relevant opt type - HostAddressOpt.

[1] I77bdb64b7e6e56ce761d76696bc4448a9bd325eb

Change-Id: I0b7b8aebc9cc70f5ea256791cfb2c61ec4357bfd
2017-03-23 15:59:16 +08:00
Felipe Monteiro 20213e0061 Adds TLS/SSL Version Support to Murano Agent.
Already implemented for Murano, but need Murano Agent
support as well.

Currently, Murano supports enabling SSL in Murano Engine [0],
but does not have a param for the user to specify the
desired SSL version. This is important because some
versions of SSL are less secure than others [1].

This patch adds ssl_version support to Murano Agent.

[0] https://docs.openstack.org/developer/murano/administrator-guide/deploy_murano/configure_ssl.html
[1] https://www.wolfssl.com/wolfSSL/Blog/Entries/2010/10/7_Differences_between_SSL_and_TLS_Protocol_Versions.html

Change-Id: I410a38d3c8294ce63c15ae98f627fa5c69a3d118
Partially-Implements: blueprint add-tls-support
Depends-On: I71c36c455cde658f402a19c59d7966cee8544cf1
2017-03-22 04:50:55 +00:00
zhurong 8739083d09 Marking rabbitmq password config property as secret
As this has changed in murano in [0],
this patch change murano-agent.

[0]: I9f41f334e0f32ee31c40c27c977ed801a68830e7

Closes-bug: #1658044
Change-Id: Idc80808dcf54eb2d457cdcf3d42e23236e07a64f
2017-01-20 10:43:29 +00:00
Artem Tiumentcev 01d6a8114f Fixed issues on Python 3
Change-Id: Ib41255ba8b2ddbe9b05c8503a90e5271ed74a7d6
2016-11-24 19:19:20 +03:00
Cao Xuan Hoang 3cfdbe8fc9 Fix LOG.warn to LOG.warning
logging.warn is deprecated in Python 3.
https://docs.python.org/3/library/logging.html#logging.warning

Change-Id: I599f637c39518076d7182b86aecbc8034076be49
2016-09-26 15:53:58 +07:00
Cao Xuan Hoang 757d5e609d TrivialFix: Remove cfg import unused
This patch removes cfg import unused in
muranoagent/tests/unit/executors/test_chef.py
muranoagent/tests/unit/executors/test_puppet.py

Change-Id: I046ece0dc5181f81b284fd362a461dfa817ff129
2016-09-09 15:15:03 +03:00
Cao Xuan Hoang f388bd73db TrivialFix: Remove logging import unused
This patch removes logging import unused in
muranoagent/executors/puppet/__init__.py

Change-Id: I09445535a42ab606ada9e8b2ca01a9bcefb4d9b1
2016-09-08 10:45:46 +07:00
Hidekazu Nakamura 1673551ce7 Chef executor unit tests now compare dicts instead of strings
Before this patch tests used to compare json.dumps representation of
dicts. This can lead to false negatives.
This patch fixes to test to compare objects instead.

Closes-Bug: #1610181
Co-Authored-By: Kirill Zaitsev <k.zaitsev@me.com>
Change-Id: I69af1c017362f95b92a81af3212f6520e7b4ceba
2016-08-05 13:52:44 +03:00
Victor Stinner 3f884f97ef Python 3: replace usage of the types module
* Replace types.DictionaryType (= dict on Python 2) with dict
* Replace types.StringTypes with six.string_types

Blueprint: murano-python-3-support
Change-Id: I2a8c61e9cd6ff15a1124b71a4e07119e2c8c30a4
2016-06-21 23:02:05 +00:00
Victor Stinner 2655a3c5a6 Python 3: only decode subprocess stdout on py2
On Python 3, stdout is already a Unicode string.

Blueprint: murano-python-3-support
Change-Id: If4b01255798689aa11fd4f512f233d9c40f102e1
2016-06-21 23:01:53 +00:00
Victor Stinner c1c400f116 Python 3: replace exec with six.exec_()
The "exec" statement was removed in Python 3. Replace it with
six.exec_() to add compatibility with Python 3.

Blueprint: murano-python-3-support
Change-Id: Idd1c6a6347e22f6a9c7887e7a327d29bb64e46d5
2016-06-21 23:00:54 +00:00
Victor Stinner c56acc78ae Python 3: Fix absolute import
Use absolute import for muranoagent.common.messaging.subscription.

Blueprint: murano-python-3-support
Change-Id: I2b70aef8fb818a841a5034154100f200513dddf7
2016-06-21 17:55:25 +00:00
Victor Stinner cdc4bde0b1 Python 3: Fix basic py3 issues
Run the sixer program to catch most common Python 3 issues and make
the code compatible with Python 2 and Python 3.

* Replace it.next() with next(it)
* Replace dict.keys()[0] with list(dict.keys())[0], since
  dict.keys() now returns an dictview on Python 3. Same change
  for dict.values()[0].
* Replace urlparse with six.moves.urllib.parse
* Replace __builtin__ with six.moves.builtins

Blueprint: murano-python-3-support
Change-Id: I9b95c3eee14c2de0010e9a3bbdffdda76496a03c
2016-06-20 21:57:12 +00:00
Lin Yang 029a2a17d7 [Trivial] Fix too many arguments for format string
Change-Id: I09afc4520b034d6b0abe2875562268272551150f
Closes-Bug: #1591614
2016-06-17 10:54:29 +08:00
Stan Lagun 7ef2eee2c3 RabbitMQ heartbeats for consumer threads
Sometimes network connection between agent and RabbitMQ
becomes corrupted without agent being aware. Usually
it happen due to some networking reconfiguration. For example
changes to iptables. Because consumer connection only reads from
the socket it cannot detect broken connection and will wait
for the message forever.

This commit enables periodic heartbeats that agent sends to the
server. If RabbitMQ doesn't receive heartbeats for some time it
disconnects the client. On the other hand because heartbeat
is a write operation it lets the client to detect broken connection
as well.

Change-Id: Id326f8d3b6dce04d83472837e11342efa8eb32fe
Closes-Bug: #1583007
Partially-Implements: blueprint rabbitmq-heartbeats
2016-05-20 21:01:37 +00:00
Stan Lagun 0efd3d5951 Allow insecure SSL communications with RabbitMQ
Add insecure option to [rabbitmq] section of config file.

Change-Id: I6ae7091df0c892ac98cd0552b1e238ad121ae6f7
Closes-Bug: #1578421
2016-05-11 11:18:30 -07:00
Stan Lagun 75712716ba Restores SIGPIPE on script processes
There is an issue https://bugs.python.org/issue1652
in Python 2.7 which was resolved in Py3.

By default Python sets SIGPIPE to SIG_IGN on startup, because
it prefers to check every write and raise an IOError exception rather
than taking SIGPIPE. This is all well and good for Python itself.
However, non-Python Unix subprocesses generally expect to have
SIGPIPE set to the default action, since that's what will happen if
they're started from a normal shell.

As a result scripts with piping that is terminated in reverse chain
order are never exit.

See more info here:
http://www.enricozini.org/blog/2009/debian/python-pipes/

Closes-Bug: #1524347
Change-Id: Ia015e7fd46912f6a0be97a6b9c297f8d14fd3f7c
2016-03-22 13:50:51 +00:00
Stan Lagun d8ece93ac5 Adds RMQ reconnection on first timeout after EP run
Execution plan can modify iptables rules causing
RabbitMQ connection to break without agent
being notified. Thus it will just not get all subsequent
execution plans.

This commit reestablishes connection upon first
5-second timeout after last executed EP so it
it broke the connection it will be fixed in
5 seconds.

Change-Id: I90c8e062be8528ffd81a295a209c3ec029fc2cd8
Closes-Bug: #1556895
2016-03-17 01:23:04 +03:00
binean fc918a2619 Fix Mutable defalut argument
Python’s default arguments are evaluated once when the
function is defined, not each time the function is called.
This means that if you use a mutable default argument
and mutate it, you will and have mutated that object for
all future calls to the function as well.

more details about this wrong usage here:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

Change-Id: Ic8eb1328852eb8a3180a08466bf18b72fd1f7956
2016-01-24 23:28:13 +08:00
Ravi Shekhar Jethani e98af85e74 Replace unicode with six.text_type
The Unicode type is 'unicode' in Python 2 and 'str' on Python 3.
This patch replaces unicode with six.text_type to make Murano
Agent more compatible with Python 3.

Blueprint murano-python-3-support

Change-Id: I03788fdfe69c16831406244fd5720a82720775cf
2015-12-29 04:15:46 -08:00
Jenkins 8c5bc5d9ec Merge "Drop bunch dependency" 2015-11-30 17:38:06 +00:00
Kirill Zaitsev dcca26bf26 Drop bunch dependency
Use custom implementation instead

Change-Id: I3c4bc932b60f1c009a0c955867cd2718d1f8ebfc
Closes-Bug: #1519841
2015-11-27 20:23:31 +03:00
Olivier Lemasle ec60c7139e Add Berkshelf support
Add Berkshelf support in murano-agent. If Berkshelf support is enabled
(Execution plan with "Type": "Chef" and "useBerkshelf": True), then the
image is expected to contain Berkshelf.

Change-Id: I036b5b9b7e1202d26fa2fd16591b680755cbfbc1
Partial-Implements: blueprint support-chef-berkshelf
2015-11-20 12:12:04 +01:00
Henar Muñoz Frutos d075358882 Remove hardcoded destinations for puppet/chef processing engine
When cookbooks are transfered from murano to
murano-agent, they are stored in another folder
which chef/puppet expect.

Closes-Bug: #1508492

Change-Id: I1363e46d655c8a641a1fa404faadc5f652f9fce4
2015-11-19 13:39:14 +00:00
Ravi Shekhar Jethani 0550e96b55 Remove unnecessary rmtree call
The clear() method is calling rmtree() to delete cache folder
but the call is being made twice. The second call is unnecessary.

Change-Id: I78b0ed2417b3775604816bf078abedce79406edd
Closes-Bug: #1513010
2015-11-04 20:04:57 -08:00
Georgy Okrokvertskhov a0c3d4f4e0 Add dynamic result message routing
This patch adds a dynamic result routing for static Agent.
This feature is designed to support Murano Agent which is
installed on external out of OpenStack system.
It adds _ReplyTo entry in the message body as well as supports
message.reply_to property.

Implements blueprint static-agent

Co-Authored-by: Alexander Tivelkov <ativelkov@mirantis.com>

Change-Id: Ief4c25a7fb63fc092f68f08b9665bde03fbe7752
2015-10-16 13:02:05 +00:00
Henar Muñoz Frutos ffcf7ec7de Adding svn for file download.
Murano agent can download files in the VM when it is executing the
 execution plan. However, just git client is allowed for download
 files. Svn client should be provided to add files in a svn repository

Closes Bug: #1486055
Change-Id: Ib1fd13fa86d012fb95d19aabda0dff7c28f99620
2015-09-23 16:25:17 +02:00
Sergey Vilgelm 910dfcf89f Remove openstack.common package
The openstack.common package is unused, so let's remove it.

Change-Id: I98f7a5be1fea63a9385c03590508da867f6a2ccb
2015-08-05 14:20:02 +00:00
Sergey Vilgelm 95b9073478 Use timeutils from oslo_utils
Remove timeutils module from openstack.common package.
Add oslo_utils to the requirements.txt.

Change-Id: Ia50402b9a379202966fb121be2c9dc698d56808d
2015-08-05 14:19:51 +00:00
Sergey Vilgelm 55f94816ea Switch to the oslo.log library
Use oslo.log instead of oslo-incubator.
Use oslo.log instead of system logging.
Add oslo.log to the requirements.txt.
Remove unused oslo-incubator modules.

Implements: blueprint port-oslo-incubator-to-oslo-log

Partial-Bug: 1481370
Change-Id: I7047497f2022b0c56be0f3811f1bb59ffe470877
2015-08-05 15:44:00 +03:00
Henar Muñoz Frutos 5fc7584583 Puppet default execution involves the definition of none modules.
The Puppet executor has been modified to consider if the recipe is empty.

Closes Bug: #1477482

Change-Id: I23aad503a758e95729b0fb3108049a6652becb32
2015-07-31 13:25:33 +02:00
Sergey Vilgelm 49c9c829b7 Switch to oslo.service
oslo.service has graduated, so murano should consume it.

Change-Id: I820a56920c1583d57880acf7d21fbd4a296ca26e
Closes-Bug: #1466851
2015-07-03 08:46:18 +00:00
Doug Hellmann f4c07e1ae7 Update requirements
Manually updating the requirements, since this project isn't in
the list getting requirements syncs from global requirements. Updating
hacking required some format changes to docstrings, and I included those
in this patch.

Change-Id: Ib33f6308ed121fa2350eb32846e476f2a9dd28f4
2015-06-25 23:15:05 +00:00